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Executive Summary 

Background 

The Internal Revenue Service (IRS), in accordance with the Office of Management and Budget’s 
(OMB) directive, launched an e-govemment initiative to improve govemment-to-business and 
govemment-to-citizen electronic capabilities. The IRS Free File program (“Free File” or “the 
program”) was one such initiative, to provide free and secure online tax return preparation and 
filing services to taxpayers. The IRS developed a solution in partnership with the Free File 
Alliance, LLC, that consists of members of the tax software industry. 

Free File provides eligible low-income taxpayers with easy access through IRS.gov 1 , which 
offers a list of all free file offerings on an IRS-hosted website. Under the agreement, Free File 
Alliance members offer both free online tax preparation and free e-filing services at no cost to 
qualifying taxpayers for federal returns. 

Agency leadership engaged MITRE to conduct an independent assessment of the Free File 
program to ensure continued operations and integrity of the program. A component of this 
assessment included gathering feedback and perspectives on the taxpayer experience with the 
Free File program to better understand how taxpayers experience the program. 

This report summarizes the findings from in-person usability testing sessions conducted July 22- 
26, 2019 as part of the Free File program assessment. Specifically, the IRS asked MITRE to 
collect information about the user experience and satisfaction finding the Free File program 
online and completing an electronic tax return through Free File software offers. MITRE 
completed 29 usability testing interviews to evaluate the taxpayer experience and provides in this 
document recommendations for improving the Free File program. 

Methods 

MITRE conducted in-person usability testing sessions with taxpayers who were identified in the 
Compliance Data Warehouse (CDW) as having submitted a tax return with the Free File program 
in Tax Year (TY) 2018. MITRE sent invitation letters to 2,500 randomly selected prior free filers 
in Chicago, IL 2 , and scheduled interviews with taxpayers who responded to the invitation. In 
total, MITRE completed 29 usability testing sessions with Chicago taxpayers. The objectives of 
these interviews were to understand how taxpayers locate the Free File program options; 
understand how taxpayers select a software provider; capture usability issues with IRS websites 
related to the Free File program; capture usability issues with specific software provider 
platforms; and understand taxpayers’ previous experiences with the Free File program. 

During the sessions, participants completed fictitious scenarios to evaluate the experiences of 
completing three tasks: 1) locating the Free File program online, 2) selecting a Free File software 
offer, including with the Free File software lookup tool, and 3) completing a tax return. MITRE 
randomly selected three Free File software offers to test with taxpayers; each participant filed a 
fictitious return with one of the three selected offers. Taxpayers used fictitious information (e.g., 
Form W-2s, Form 1098-E, and other documentation) to complete the tasks. MITRE researchers 


1 https://www.IRS.gov/ 

2 The MITRE research team selected the Chicago, IL location for two reasons: 1) sufficient volume of known free 
filers within a geographical area; 2) familiarity with the location due to having conducted research on the redesigned 
1040 Form in October, 2018, with a distinct population of taxpayers (taxpayers who filed on paper). 
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observed the sessions and then asked debriefing questions to establish what aspects of the Free 
File system (both the IRS.gov Free File pages and the specific Free File software offer web 
pages) caused confusion, errors, or frustration. In addition to data on the usability of the Free File 
system, MITRE also collected and analyzed data on taxpayer satisfaction and experience (via 
brief questionnaires), and taxpayers’ perceptions of the Free File system components and the 
Free File program. 

Findings 

MITRE identified and reported all usability issues observed by the research team with each of 
the three Free File usability tasks and provided suggested solutions for addressing each of these 
issues individually. Drawing on the data collected from the usability tasks, analysis of qualitative 
data gathered during debriefings and interview questions, the satisfaction and user experience 
questionnaires, and metrics on taxpayer behaviors and selections during the tasks, MITRE 
identified the following high-level findings: 

• Taxpayers lack understanding of what the Free File program is and its relationship 
to the IRS. Despite the fact that MITRE specifically recruited taxpayers recorded in CDW 
as having previously used Free File, many participants conflated the Free File program 
with other forms of filing taxes for free, including free commercial software, and were 
unsure whether they had used Free File or a free commercial version in the past. Some 
reported not having heard of Free File until the past year. Taxpayers also seemed unclear 
on the relationship between the IRS and Free File. A few never sought out IRS.gov and 
did not seem to realize that they should use IRS resources to find the program, whereas 
others expressed surprise that Free File offers were all provided by third-party tax 
software providers and not directly from the IRS. 

• Taxpayers struggled to find the Free File landing page, including through IRS.gov. 

Most taxpayers approached finding Free File by starting with a search engine query, using 
keywords such as “IRS,” “free tax filing,” and “free file.” This suggests that the IRS needs 
to plan for taxpayers to enter the Free File system from search query results, rather than 
from the IRS home page. Some ended up on commercial software provider websites; most 
were able to eventually find the Free File landing page on their own, though through 
various routes that caused confusion. Some taxpayers meandered through pages of 
IRS.gov looking for Free File and expressed frustration that they could not find it. 

• Taxpayers experienced difficulties and confusion with selecting a software offering, 

particularly when evaluating Earned Income Tax Credit (EITC) eligibility. 

o EITC eligibility requirements were confusing for taxpayers and a substantial 
proportion (about one-third) committed errors when selecting a Free File software 
offer using the lookup tool due to erroneously answering the EITC eligibility question. 
Information on IRS.gov about EITC eligibility largely did not resolve this problem. 

o Common issues taxpayers had with the lookup tool included not noticing the lookup 
tool, not understanding its purpose, and committing errors on EITC eligibility. 

o When taxpayers commit errors using the lookup tool, the results could include Free 
File software offers taxpayers may not be eligible for; taxpayers might only discover 
this down the road when they are midway through completing a tax return. Incorrectly 
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selecting “yes” for EITC eligibility in particular causes more results returned from the 
lookup tool than are accurate. 

Taxpayers expressed that they were overwhelmed with choices and information. 

Many taxpayers stated that there were simply too many options and too many eligibility 
criteria to digest and understand. This led to taxpayer frustration and to taxpayers having 
difficulty selecting a software offering, both on the Free File software offers page and, in 
the results, returned from the lookup tool. 

Taxpayers used strategies beyond eligibility criteria to select software offers. The 

most co mm on reasons taxpayers selected a particular offer were prior experience with a 
tax software provider and name recognition. Some taxpayers expressed that they needed to 
rely on rationale beyond eligibility criteria to choose a Free File software offer because 
there were so many choices. This has the potential for taxpayers to commit errors and 
select Free File software offers for which they are not eligible, if criteria such as name 
recognition override the current year’s eligibility criteria. 

Common usability issues with the selected tax preparation software programs 
included trouble finding where to report adjustments to income, difficulty choosing 
between filing statuses, and trouble navigating different systems’ flow. Taxpayers 
often didn’t realize that options selected at various points in the return preparation process, 
such as selecting checkboxes up front to indicate what types of adjustments and 
deductions they had, would dictate their ability to enter those amounts in later. This was 
relevant for the student loan interest deduction included in the test scenario. Additionally, 
when filing status was not clear cut (as in the test scenario), participants had trouble 
getting the information they needed to determine filing status correctly; some systems did 
not walk taxpayers through the necessary steps. Additionally, various aspects of the flow 
of tax preparation systems caused confusion. Some taxpayers had trouble navigating 
between different components of the system. The “Quick File” option in one system 
caused major problems when taxpayers found that they did not have the tax knowledge to 
fill out their forms a la carte. 

Taxpayer ratings of ease and satisfaction varied by task, software provider, and 
demographic characteristics. Taxpayers rated finding Free File and selecting a Free File 
software offer as acceptably easy tasks, whereas filing a return was rated as more difficult. 
Moreover, ease of completing a return was greater with some Free File software offers 
than others, and the results indicate that struggling with a tax software provider is 
associated with a lower overall satisfaction rating on the Free File system as a whole. This 
suggests that taxpayer experience with a particular Free File member program could color 
taxpayer perceptions of the entire Free File experience. Examining patterns in the small 
sample in this study, there is some evidence that Free File usability is lowest for taxpayers 
in the upper two age terciles represented in this study (over age 35). 
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Recommendations 

MITRE provides the following issue-level recommendations: 

• Standardize how Free File is referenced on IRS platforms and communications. 

Currently, different web pages at IRS.gov have different labels and titles for Free File, 
such as buttons that say, “File Your Taxes for Free” versus “Free File.” Standardize all 
references to one label or title and use that label/title in all communications about the 
program. Encourage Free File members to follow suit. This should also help taxpayers 
access Free File from web searches, as it may influence their search terms. 

o Clarify the IRS’s relationship to the Free File program and the software offers 

o Use visual cues like third-party logos on the IRS site to signal that taxpayers will go to 
the software provider website 

o Improve taxpayers’ ability to find the Free File landing page both from IRS.gov web 
pages (e.g., the home page) and directly from internet search results 

• Take steps to increase ease of understanding eligibility requirements. The current 
volume and complexity of eligibility requirements may lead to taxpayer mistakes in 
selecting a Free File software offer. Clearer, simplified eligibility requirements will help 
taxpayers focus on the important information to choose a Free File software offer. 

o Reduce the amount of information presented in individual boxes on the software offers 
page. 

o Consider presenting eligibility criteria on the software offers page in a new format, 
such as in a grid view where all criteria are presented consistently, or via tooltips. 

o Include information on eligibility criteria in the results returned by the software lookup 
tool. 

• Ensure that taxpayers understand that determining eligibility for a software offer is 
ultimately their responsibility. Taxpayers expressed the belief that their eligibility 
would be ‘checked’ at some point along the way, for example, by the software lookup 
tool. MITRE recommends that the IRS incorporate language on the Free File pages and 
lookup tool to help taxpayers understand that misunderstanding or misreporting eligibility 
criteria can lead to their selecting a Free File software offer for which they are not 
eligible. 

• Eliminate or change the EITC eligibility question in the lookup tool and on the 

software offers page. If the IRS stated on the Free File site that all taxpayers eligible for 
EITC are eligible for Free File, the question could be removed from the tool. MITRE 
recognizes this would be a topic for negotiation between the IRS and the Free File 
Alliance, LLC (FFA) because of how that could impact some companies’ appearance in 
the tool. If the IRS chooses to leave EITC in the lookup, MITRE recommends linking the 
question to the EITC eligibility tool, but ONLY after conducting usability testing on the 
EITC eligibility tool and addressing the results. 

• Recommend suggested solutions be shared with FFA to increase usability of 
members’ tax preparation software systems. MITRE recommends that the IRS share 
the FFA member website suggested solutions within this report and findings identified in 
the taxpayer journey study with the FFA. This can help member programs increase the 
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likelihood of taxpayers having a positive experience with Free File by improving look and 
feel, communication of eligibility on member home pages, and how to best walk taxpayers 
through their online tax return as they fill it out. Recognize that the experience of using a 
particular member offer will affect how taxpayers perceive the Free File experience as a 
whole. 

Conduct research on the usability taxpayer access and use of the Free File system. 

MITRE recommends that the IRS and Free File members conduct iterative testing to 
evaluate system design and usability. MITRE recommends that this usability testing 
include representation of different age groups, income groups, and education groups, to 
ensure that the Free File system is maximally usable for all eligible taxpayers. 

Conduct research on Free File awareness, attending to different demographic 
groups’ awareness. The present study was not designed to assess awareness of Free File 
among the taxpayer population. MITRE recommends that the IRS conduct future research 
to better understand whether taxpayers are aware of Free File, their understandings of 
what the program is, and their reasons for using or not using Free File. MITRE 
recommends including both previous free filers and eligible taxpayers who have not 
previously used Free File in this research. 
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1 Introduction 

This report details the findings from MITRE’s usability study of the Internal Revenue Service 
(IRS) Free File program, including Free File-related websites and three randomly selected 
Free File member software offers. This study is part of MITRE’s independent assessment of 
the Free File program to evaluate various facets of the program, including Free File Alliance 
(FFA) member compliance with the Memorandum of Understanding (MOU), taxpayer 
patterns in Free File eligibility and use, and design of Free File software providers’ systems. 

The scope of this report is solely the results of the taxpayer usability testing, in which MITRE 
conducted 29 usability interviews with previous free filers in Chicago, IL. These testing 
sessions observed usability issues and collected taxpayer feedback on the experiences of 
locating Free File, selecting a Free File software offer, and completing a return using one of 
three randomly selected member offers. 

The IRS, in accordance with the Office of Management and Budget’s (OMB) directive, launched 
an e-govemment initiative to improve govemment-to-business and govemment-to-citizen 
electronic capabilities. The IRS Free File program (“Free File” or “the program”) was one such 
initiative, to provide free and secure online tax return preparation and filing services to 
taxpayers. The IRS developed a solution in partnership with the Free File Alliance, LLC, that 
consists of members of the tax software industry. Free File provides eligible low-income 
taxpayers with easy access through IRS’s website 3 to Free File offerings from commercial tax 
software developers, via a list of Free File software offers on an IRS-hosted website. Under the 
agreement, Free File Inc., members offer both free online tax preparation and free e-filing 
services. There is no cost to qualifying taxpayers for federal returns. 

Agency leadership engaged MITRE to conduct an independent assessment of Free File to ensure 
continued operations and integrity of the program. A component of this assessment included 
gathering feedback and perspectives on the taxpayer experience with Free File to better 
understand how taxpayers experience the program. 

The scope of the task included conducting in-person testing sessions with taxpayers who 
previously used Free File to better understand how taxpayers experience the program. This 
document reports the findings from the 29 testing sessions conducted in July 2019. This includes 
research objectives, overall findings, and recommendations, methodology, specific usability 
issues, and suggested solutions for fixing them. 


3 https://www.IRS.gov/ 


1 

PRE-DECISIONAL - FOR INTERNAL IRS W&l USE ONLY - NOT FOR PUBLIC RELEASE 



PRE-DECISIONAL - FOR INTERNAL IRS W&l USE ONLY - NOT FOR PUBLIC RELEASE 


For the purposes of this report, we define the Free File system as including the following 
components. The taxpayer testing sessions touched on these components: 

1. Navigating the internet to locate Free File program information and options 

2. IRS.gov web pages related to the Free File program, 4 including the Free File landing 
page, the Free File software offers page that lists tax software offers provided by FFA, 
and the Free File software lookup tool 

3. Filing a federal tax return with one of three randomly selected Free File software offers 

1.1 Research Objectives 

The purpose of the taxpayer research study was to test the taxpayer experience with the IRS Free 
File program and its online components. Specifically, the research conducted for this study met 
the following objectives: 

• Understand how taxpayers locate the IRS Free File program 

• Understand how taxpayers identify and select Free File member programs to use for tax 
filing 

• Understand taxpayer perceptions of the IRS Free File software lookup tool 

• Identify potential issues with taxpayers’ ability to locate and use the IRS Free File 
program, including issues with the following: 

o How taxpayers determine which member software offer to use 

o Which member programs taxpayers select 

o What issues taxpayers encounter when completing an online tax return using a selected 
member’s program 

o Where taxpayers would seek information or assistance using Free File program 
components 

• Measure taxpayer satisfaction with the IRS Free File program, including ease of 
completing Free File system-related tasks 

1.2 Scope 

This report describes the results of 29 in-person interviews conducted with taxpayers in six zip 
codes in Chicago, IL, who had used the Free File program in Tax Year (TY) 2018. The 
interviews collected four types of data to meet the research objectives: 

1. Usability data, gathered by observing taxpayers locate the IRS Free File program, select a 
Free File software offer, and complete an online return using a Free File member’s 
program 


4 Pages that taxpayers may have interacted with during testing sessions included, but were not limited to: 
https://www.irs.gov/filing/free-file-do-your-federal-taxes-for-free, https://apps.irs.gov/app/freeFile/, and 
http s: // app s. ir s. go v/app/fr e eF ile/j sp/wizard.j sp ? 
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2. Metrics on taxpayer activities in the testing session, including success rates at finding the 
IRS.gov Free File program web pages and selecting a Free File software offer 

3. Taxpayer ratings of user experience and satisfaction using the Free File program 

4. Qualitative data on taxpayer perspectives on the Free File program, Free File member 
offers, and previous Free File experiences 

The MITRE team, in collaboration with the sponsor, decided to interview previous free filers 
because these taxpayers are identifiable in the Compliance Data Warehouse (CDW), allowing 
MITRE to contact them easily. Restricting the sample to previous free filers also ensured that 
participants fell within the target population of taxpayers who would seek free tax filing services, 
enhancing the validity of the testing results. The MITRE research team selected Chicago, IL for 
two reasons: 1) sufficient volume of known free filers within a geographical area; and 2) 
familiarity with the location due to having conducted research on the redesigned 1040 Form in 
October 2018, with a population of taxpayers who filed on paper. 

1.3 Study Limitations 

This study has several limitations that may affect the external validity, meaning the extent to 
which the results of this study can be generalized beyond the context of the study setting. 

• Artificial testing situation. When completing most of the testing activities, including 
filling out a tax return, taxpayers did not use their real, personally identifying 
information. Although this was necessary to protect privacy and confidentiality, it 
resulted in an artificial testing situation. This had two potential impacts on the 
interpretation of the study’s results: 

o Artificial scenarios may have been difficult to understand for taxpayers, introducing a 
layer of cognitive burden that differed from a real-life tax reporting situation. To 
mitigate this limitation, MITRE researchers pre-tested the scenarios to make sure they 
were not too burdensome or unclear to the point of hindering the usability test. MITRE 
interviewers were trained to be attentive to signs of burden or fatigue and end the 
interview if participants were no longer providing quality data. 

o The artificial testing situation required participants to report tax data elements that may 
not have been relevant to their real-life situation (e.g., student loan interest deduction). 
This potential lack of knowledge could have been a confounding factor and could have 
caused errors. To mitigate this limitation and the challenge it posed for external 
validity, MITRE conducted an analysis of Free File program users reporting behaviors 
from TY 2017 and TY 2018 to tailor the scenarios to reflect “typical” reporting. 

• Characteristics of the interview sample may not perfectly mirror Free File program 
users nationally. Demographic characteristics of prior free filers vary by geographic 
location and urbanicity. MITRE selected the Chicago, IL area based on sufficient volume 
of free filers and convenience of the location. However, Chicago free filers tend to be 
slightly younger and a greater proportion filed as single compared to free filers nationally 
(see Section 2.2). Additionally, MITRE was unable to control which taxpayers had the 
interest and availability to contact MITRE to participate in an interview. The resulting 
demographic characteristics of the interviewed sample therefore do not exactly mirror the 
characteristics of the Free File program user population nationally, simply because some 
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taxpayers were more likely to call MITRE wanting to participate than others. As such, 
caution should be used when generalizing from the sample of taxpayers interviewed here 
to the full taxpayer population. 

Time constraints. This study was completed in six weeks to support the broader 
assessment timeline. This time constraint cut back on the time available to MITRE 
researchers to pre-test and refine the testing protocol. It also necessitated that MITRE 
visit an interview site previously known to the team from a prior study to reduce the time 
spent planning logistics. 

Influence of language used in recruitment and interview materials. To clearly 
communicate the purpose of the study and its legitimacy as sanctioned by the IRS, 
recruitment and interview materials included references to the IRS Free File program 
specifically. This may have influenced taxpayers’ behavior in the usability testing 
interviews, particularly when they were tasked with locating where they would file their 
federal return for free. MITRE captured all search terms from the usability sessions to 
assess the extent to which they mirrored language used by MITRE research materials. 

Cannot generalize findings to new Free File taxpayers. This study only included prior 
Free File program users; therefore, the findings cannot necessarily be used to assess how 
new users might experience the Free File program. 

Cannot validate prior Free File eligibility of participants. MITRE researchers had two 
data points to attempt to verify prior Free File use and eligibility: the Free File indicator 
available in CDW and potential participants’ own reports of prior Free File use during 
eligibility screening. However, some records from CDW indicate that taxpayers 
identified as having used Free File also had ineligible (high) Adjusted Gross Income 
(AGI), suggesting that these taxpayers may not have actually been eligible for the 
program. MITRE cannot confirm or deny these taxpayers’ previous eligibility and doing 
so is outside the scope of this study. 

MITRE’s results reflect the state of the web pages and member software systems on 
the testing dates. MITRE’s in-person testing was done July 22-26, 2019. Changes to the 
IRS.gov pages or member sites done before or after those dates are not in scope of this 
study. 
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1.4 Organization of This Document 

This document is organized into the following sections, shown in Table 1. 


Table 1. Document Organization. 


Section 

Contents 

1. Introduction 

Describes the research objectives and scope of this study. 
Defines this document’s purpose, scope, and organization. 

2. Methods Overview 

Provides a high-level overview of the methodological 
approach to this study. 

3. Study Findings and 
Recommendations 

Provides a summary of the high-level findings with 
corresponding evidence generated from this study and 
recommendations for the IRS. 

4. Usability Findings 

Communicates the usability issues identified in the taxpayer 
testing sessions. 

5. User Experience 

Questionnaire Findings 

Communicates the responses on the user satisfaction and 
experience questionnaire. 

6. Debriefing Findings 

Communicates themes MITRE identified from the interview 
debriefing questions on taxpayer perceptions. 

7. Taxpayers Pathways 
and Metrics 

Communicates the users flows and pathways taxpayers 
followed to locate Free File and metrics regarding taxpayer 
behaviors while completing the usability tasks, including 
errors committed and offer selections. 

8. Conclusion 

Describes MITRE’s high-level conclusions based on the study 
findings. 

9. References 

Provides works cited throughout the text. 

Detailed Methods 
Appendix 

Provides detailed accounting of the study methodology, 
including sample design and selection, interview methodology 
and procedures, and analytical approach. 

Usability Issues Master 

List Appendix 

Compiles a full list of all identified usability issues into a 
single table for easy reference. 

Other Appendices 

Attaches key documents, such as recruitment materials, the 
scenarios, and other interview materials. Also includes 
detailed tables of free filer characteristics used in the 
development of the sampling strategy. 
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2 Methods Overview 

This section provides an overview of the approach taken through the various phases of 
the research process. It also provides a closer look at the scenarios and corresponding 
fictitious tax information the team developed in order for the taxpayer participants to 
complete the tasks in the usability sessions. The population of interest was prior Free 
File program users. 

• Sampled taxpayers received letters from the IRS and MITRE about the study, inviting 
them to contact MITRE to participate. 

• Interviewers collected data on usability, taxpayer perceptions of the Free File program, 
prior Free File experiences, as well as taxpayer satisfaction and taxpayer ratings on the 
ease of completing Free File-related tasks. 


2.1 Overview of Research Process 

The goals of the IRS Free File usability test were to identify taxpayer issues with: 

• Locating the IRS Free File program website using commonly available search engines 

• Identifying Free File software offers for which they may be eligible, including use of the 
Free File software lookup tool 

• Filing a return using a Free File software offer selected at random by MITRE in advance 
of the testing session 

For this study, taxpayers participated in an in-person session in which they completed the three 
tasks listed above, answered debriefing questions about their experiences completing the tasks 
and their prior Free File experiences, and completed two brief questionnaires measuring the ease 
of completing the tasks and overall taxpayer satisfaction with the Free File system. MITRE 
conducted 29 in-person interviews from July 22-26, 2019. Taxpayers used the Free File web 
pages and Free File member systems as they existed on those dates. 

MITRE randomly selected three Free File program members’ software offers to test during the 
usability sessions. These Free File software offers were selected to represent the spectrum of 
complexity of eligibility criteria used by the FFA members providing Free File software offers. 
See section 2.4 for a full description of how MITRE selected three Free File software offers to 
test. 


2.2 Recruitment 

The goal of this study was to gather usability information from a sample of prior IRS Free File 
program users that reside in a metropolitan region with sufficient free filers to result in 25 to 30 
study participants. Given the schedule limitation, the team initiated its sampling strategy by 
leveraging taxpayer data from the CDW within the zip code clusters used in MITRE’s usability 
test to support the Form 1040 redesign independent assessment completed in May 2019 [1]. 
MITRE examined the volume of TY 2017 and TY 2018 free filers located in the Chicago, IL, 
Portland, OR, Dallas TX, and Pittsburg, PA regions, used for the previous Form 1040 redesign 
assessment, to determine whether any of these regions would be an appropriate location for the 
Free File taxpayer testing as well. Note that the populations sampled in the Form 1040 redesign 
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study and this Free File study are entirely distinct—prior free filers, are, by definition, electronic 
tax filers, whereas the Form 1040 assessment sampled prior paper filers. Thus, there is minimal 
risk of overlap in the two target populations. 

The total universe of free filers in TY 2018 was 2,343,235. 5 After an initial exploration of the 
volume of free filers in multiple metropolitan areas nationally, MITRE identified Chicago, IL as 
the best candidate location for the taxpayer interviews. Within the Chicago, IL cluster, the team 
identified 4,106 taxpayers who were identified in CDW as having used Free File in TY 2018. 

The zip codes include in this cluster are 60657, 60640, 60618, 60613, 60625, and 60660. From 
this pool of 4,106, MITRE randomly sampled 2,500 taxpayers to receive invitation letters. The 
following steps were taken to identify taxpayers for testing: 

1. Data were cleaned to remove duplicate Taxpayer Identification Numbers (TINs), TINs 
with missing zip codes, TINs with international zip codes, TINs missing a software 
vendor code, TINs with rejected returns, and taxpayers under 18 years of age. 

2. Randomly sampled 2,500 returns from the six Chicago zip codes. 

3. IRS sent recruitment letters to the 2,500 individuals to notify recipients that MITRE will 
be conducting a study to better understand how taxpayers interact with the IRS Free File 
program and that they will be receiving an invitation letter from MITRE. 

4. IRS sent the MITRE invitation letter to participants. This letter explained the purpose of 
the study; how to contact MITRE to participate; compensation that taxpayers will receive 
for participating; and how to verify that this study is legitimate. Both letters are in 
Appendix C. 

MITRE had 2,500 invitation letters mailed and received responses from 141 potential 
participants. Overall, 5.64 percent of taxpayers contacted MITRE to express interest in 
participating in the study, either via phone call or via the online form hosted at www.mitre.org. 
Additionally, MITRE created a waitlist for participant interviews for each site in case a 
minimum necessary number of interviews could not be conducted due to respondents’ failure to 
appear. Two participants failed to appear to their scheduled interview, and one participant 
cancelled approximately 24 hours in advance of their scheduled interview. MITRE called two 
candidates from the waitlist participants. Twenty-nine total interviews were deemed 
“completed,” meaning that the participants were eligible, and testing took place. 

The vast majority (92 percent) of respondents filed as single, and the largest age range was 31-55 
years of age (53 percent) followed by 18-39 (28 percent). Forty-two percent reported a 
household income between $15,001 and $35,000 and the most common educational level was a 
bachelor’s degree (53 percent) followed by having a master’s degree (28 percent). Note that 
participant recruitment ultimately depended upon the taxpayers themselves contacting MITRE 
after receiving an invitation letter; therefore, MITRE was not able to dictate the characteristics of 
the taxpayers who participated beyond the sample plan described in this section. 


5 These data were drawn from the CDW (ETARASMEF1544 table), with ENGLISH_FREE_FILE_RET_IND=1, 
accepted_rtn_ind=l, and sofitware id number not null. The data were cleaned to remove duplicate taxpayer 
identification numbers, records with international zip codes (zip=0), records with missing zip code, and taxpayers 
under age 18. 
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Letter recipients called MITRE or visited a MITRE web page to give their contact information 
and express interest in participating. MITRE spoke with interested letter recipients on the phone 
to verify eligibility and ask basic questions. During the screening phone call, respondents were 
deemed ineligible if they: had not received the invitation letter, do not live in one of the six zip 
codes, or reported not having used the IRS Free File program in the past two years. If a taxpayer 
indicated they did file a tax return for free within the past two years, but were uncertain if their 
return was file through IRS Free File or a free filing product offered by a commercial tax 
software provider, they were considered eligible for this study, provided they satisfied the other 
screening criteria. Those who met the screening criteria were asked additional questions to 
capture demographic information. 

Participants were asked: 

• Within the last two tax years (2017 and 2018), have you filed your federal tax return 
through the IRS Free File program? 

• Would you say that you are the adult in your household who is most familiar with the 
preparation and filing of your federal income tax returns? 

• What was your filing status for tax year 2018? 

• What is your age range 6 ? 

• Which is your range of household income? 

• What is the highest degree or level of school you have completed? 

• Did you or your spouse receive military pay in 2018? 

2.3 Overview of Testing Scenarios 

In preparation for the usability interviews, the MITRE team developed three fictitious scenarios. 
Two scenarios were used to support tasks where taxpayers were asked to select a Free File 
software offer for which a fictitious person was eligible (selection scenarios 1 and 2). In the third 
scenario taxpayers were asked to complete a tax return using a Free File software offer chosen by 
MITRE (filing scenario). To design the selection scenarios, MITRE examined eligibility criteria 
for different Free File software offers and developed taxpayer characteristics that would return 
different sets of possible Free File software offers. To design the filing scenario, the team had 
two goals: 1) to introduce enough complexity into the scenario so that taxpayers would have to 
interact with multiple components of the tax preparation software, including filing status, 
dependents, and adjustments to income, while still keeping the scenario simple enough to 
complete in approximately 30 minutes, and 2) to ensure that scenarios represented feasible free 
filer filing characteristics. Using CDW data, the team identified which fields on previous returns 
were commonly populated by individuals filing through Free File. In particular, MITRE’s review 
of the free filer data available in CDW shows (see Appendix A for detailed descriptive statistics): 


6 When screened, participants were asked to select from a set of age ranges. Specific ages were requested for 
participants who participated in the study. 
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• 71 percent of taxpayers who filed through the IRS Free File program in TY 2018 reported 
a filing status of single; the second most co mm on filing status was head of household (17 
percent) 

• Mean AGI for free filers in TY 2018 was $23,247 

• Thirty-five percent of taxpayers using the IRS Free File program in TY 2018 claimed any 
non-zero amount for Earned Income Tax Credit (EITC), with a mean EITC distribution 
value of $566 

• Of the taxpayers using the IRS Free File program in TY 2018, 1.6% itemized deductions 

Based on these demographics, the MITRE team developed three testing scenarios for participants 
to perform. All scenarios used a fictious name, address, and other taxpayer information. Any 
resemblance to actual persons, living or dead, is purely coincidental. A summary of each 
scenario is provided below. See Appendix A for the complete set of scenarios used during the 
usability study. 

• Selection Scenario 1. Participants used this scenario to identify and select a Free File 
software offer using the IRS Free File program web page (which may have included using 
the software lookup tool). The scenario contains details about a single taxpayer, age 52, 
who resides in Massachusetts, has one dependent, and 2018 AGI of $49,000. This scenario 
was designed to return eligibility results that do not include the large company software 
providers, which together represent approximately 70 percent of Free File returns filed in 
TY2018 7 . 

• Selection Scenario 2. Participants also used this scenario to identify and select a Free File 
software offer using the IRS Free File program web page (which may have included using 
the software lookup tool). The scenario contains details about a single taxpayer, age 43, 
who resides in Texas, has no dependents, and 2018 AGI of $33,000. This scenario was 
designed to return eligibility results that do not include options to file a state return for 
free. 

• Filing Scenario. Participants used this scenario to complete an online federal tax return 
using a Free File software offering selected at random by MITRE. The scenario contains 
details about a single taxpayer with one dependent and wage income from one Form W-2, 
and student loan interest paid as reported on Form 1098-E. This taxpayer is also eligible for 
the EITC. 

Selection scenarios 1 and 2 were required to complete the tasks related to the Free File software 
lookup tool. To control for order effect, MITRE alternated the order that participants received 
selection scenarios 1 and 2. All participants completed the same filing scenario. 


7 Based on data from CDW. 
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2.4 Selection of IRS Free File Member Programs for Testing 

The scope of this study was to conduct 25-30 in-person usability testing sessions with prior free 
filers. Because of this relatively low sample size, MITRE determined that testing three member 
tax preparation software systems was feasible. Each participant had time to complete a tax return 
with one system. MITRE devised a method for selecting three member systems at random that 
accounts for the share of free filers in TY 2018 that used certain Free File software offers, and 
the complexity of eligibility criteria. 

Specifically, MITRE’s selection method operationalized: 

• Size—in terms of percentage of free filers served (see Appendix D, Table 78 for the 
percentage of taxpayers using each member offer in TY 2018) 

• Complexity—operationalized as the number of bullet points used to describe eligibility 
for the service on the IRS site. In July 2019, the number of bullet points for a given Free 
File software offer ranged from two to six. Examples of how these bullets are displayed 
on the Free File software offers web page are shown in Figure l. 8 


eSmart Free File Edition 1Q40NQW.NET 

Receive a free Federal return if: Receive a free Federal return if: 

If your Adjusted Gross Income (AGI) is $66,000 or less, AND 
You live in one of the following states: AL, AR, AZ, CA, GA, IA, 

ID, IN, KY, Ml, MN, MO, MS, NC, ND, NY, OK, OR, Rl, SC, VA, VT or 
WV 

OR 

Your AGI is $66,000 or less, AND you are 60 or younger, AND 
You live in one of the following states: CO, CT, DC, DE, HI, IL, 

KS, LA, MA, MD, ME, MT, NE, NJ, NM, OH, PA, UT or Wl 

OR 

Free for Active Military for Adjusted Gross Income of $66,000 
or less 

Available for foreign address filers 

Figure 1. Eligibility Criteria Bullets on Free File Software Offers Page 
(https ://apps.irs.gov/app/freeF ile/). 

In order to choose three Free File software offers, MITRE split all Free File software offers into 
the categories, and randomly selected one within each category. Selection was done on July 3, 
2019, and thus represents the state of the Free File software offers page on that date. 

1. Category 1: Top two Free File software providers by percentage served. Rationale: These 
providers dominate the market, and MITRE received information from the sponsor that 
these companies have their own usability testing operations, which suggests that there 
may be fewer usability problems for taxpayers using these suppliers. 

2. Category 2: Not the top two Free File software providers by percentage served AND has 
fewer than five eligibility bullets. 


Your Adjusted Gross Income is $66,000 or less, AND your age is 
53 or younger, OR 

Free for Active Military for Adjusted Gross Income of $66,000 
or less 


8 Eligibility criteria boxes represent the state of the Free File software offers page on August 25, 2019. 
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2. Category 3: Not the top two Free File software providers by percentage served AND has 
five or more description bullets. 

Random selection was done using a random number generator in Microsoft Excel. The selected 
Free File software offers were Member 10, Member 11, and Member 3. Table 2 shows the tested 
software offers and the number of interviews completed using each offer. 

Table 2. Count of Selected Free File Software Offers Used in Interviews. 


Software Offer 

Count 

Member 3 11 

Member 11 

9 

Member 10 

9 


2.5 Interview Process 

Interview sessions were in-person and lasted no more than 90 minutes. For each session, two 
MITRE staff were present—a facilitator and a notetaker. Each session had the following format: 

1. Introduction and consent form 

2. Task 1: Free Search Engine Testing 

3. Task 1: Debrief (Single Ease Question [SEQ] and follow-up questions) 

4. Task 2: Free File software lookup tool Testing 

5. Task 2: Debrief (SEQ and follow-up questions) 

6. Task 3: Filing with Select Vendor 

7. Task 3: Debrief (SEQ and follow-up questions) 

8. Usability Metric for User Experience (UMUX) Questionnaire 

9. Final Debriefing Questions 

10. Honorarium and interview closeout 

2.5.1 Introduction and Consent 

During the introduction, the facilitator informed the participant about the study’s purpose and 
format. In order to solicit unfiltered perceptions, participants were also told that their information 
would be kept confidential, that there were no right or wrong answers, MITRE had no 
commercial interest in the IRS Free File program, and that MITRE wanted their candid feedback. 
Participants were also informed that, with their consent, an audio recording of the session would 
be created in addition to recording of the participants online activity with Morae usability 
software developed by TechSmith Corporation. Next, participants read and signed a consent 
form that informed them of their rights as a participant. 
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2.5.2 Task 1 Search Engine Testing and Debriefing 

To complete the first task, participants were provided with a computer that had access to the 
internet. The computer’s desktop screen contained shortcuts to Chrome, Firefox, and Internet 
Explorer browsers. Figure 2 provides a screen capture of the desktop view. Participants were 
asked to use the computer to locate the program that allows them to file a federal return for free. 9 * * 12 
MITRE researchers allowed participants to continue as far as selecting a Free File software offer, 
if they were successful at locating the IRS.gov Free File landing page 
(https://www.irs.gov/filing/free-file-do-your-federal-taxes-for-free) independently with no 
intervention from the researcher. If participants arrived elsewhere (e.g., a commercial tax 
software provider’s site) or were unable to find Free File, the researcher eventually stepped in 
and directed the participant to IRS.gov to look there. Thus, some participants selected a Free File 
software offer that they themselves would be eligible for based on their actual characteristics, 
whereas others did not complete the task. 

At the conclusion of the task, participants were issued a SEQ to rate the ease of completing the 
task of navigating the internet to file a federal return for free. The researcher then asked 
debriefing questions to better understand the participants’ processes and thoughts on the task. 


« 0 

Recycle Bin Morat 

Manager 

J- 0 



Figure 2. Screenshot of Computer Desktop Used for Testing. 


2.5.3 Task 2 Free File Software Lookup Tool Testing and Debriefing 

The participants were issued the first scenario (selection scenario 1 or 2) and asked to use the 
scenario to identify and select a Free File software offering, starting from the IRS Free File 
program landing page (https://www.irs.gov/filing/free-file-do-your-federal-taxes-for-free). If the 
participant did not elect to use the Free File software lookup tool to complete the task, they were 


9 The language used to introduce this task in the protocol was, “For your first activity, imagine that you have learned that there is 

a program available through the IRS that allows you to file your federal tax return for free. Our goal is to understand how you 
would go about locating this free program.” Although the term “IRS Free File program” was used in the introduction to 

the session (explaining the purpose of the research), the MITRE team avoided reusing these terms while introducing 
the tasks in order to reduce influencing the participants’ search terms. 
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asked to repeat the task using the lookup tool after completing it without the tool. This allowed 
MITRE researchers to both observe what participants would do on their own, and to also test the 
lookup tool. Participants were then issued the alternate selection scenario and asked to identify 
and select a Free File software offering by using the lookup tool. 

At the conclusion of the task, participants were issued the SEQ to rate the ease of using the 
software lookup tool and asked debriefing questions about their experience identifying and 
selecting a Free File software offering. 

2.5.4 Task 3 Filing with Selected Vendor and Debriefing 

The participants were issued the filing scenario and asked to use one of the three selected Free 
File software offers to complete an online federal tax return. Participants were asked to complete 
the return up to, but not including, submission of completed return. 

At the conclusion of the task, participants completed a final SEQ and answered debriefing 
questions about their experience completing an online federal tax return using the selected 
software offering. 

2.5.5 Final Questionnaire and Honorarium 

Participants completed a four-question user experience questionnaire (UMUX-Lite) and 
answered more general questions about their prior experience with the IRS Free File program 
and their suggestions to make the program easier for taxpayers. Participants then received a $100 
honorarium and were thanked for participation. 

During the session, the notetaker observed the participant and recorded points at which the 
participant appeared to be confused, anything the participant verbalized, and certain pre¬ 
determined behaviors of interest. The notetaker used a pre-prepared notes template for each 
portion of the interview. This template enabled notetakers to capture observation notes as well as 
participant responses to the debriefing questions. 
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3 Study Findings and Recommendations 

This section provides a summary of MITRE’s findings and recommendations derived from all 
data collected in the taxpayer testing sessions. Specifically, MITRE draws on: 1) the usability 
findings describing issues that the team identified in the in-person testing sessions; 2) 
qualitative data gathered during debriefing questions in the interview; 3) metrics on taxpayer 
behaviors for specific items related to locating and using the IRS Free File Program, as 
collected in the testing sessions; 4) findings of the satisfaction and user experience ratings 
collected via the SEQs and the UMUX-Lite questionnaires. 

In this findings-recommendations format, MITRE uses the following naming conventions to 
organize the results: 

• Findings: Information derived from the findings collected in the interviews that is 
relevant to the scope of the assessment, synthesized to draw conclusions from the 
evidence. 

• Recommendations: Concrete actions that the IRS or software providers can take to 
improve the Free File system or better support taxpayers. 
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Table 3. Taxpayer Lack of Awareness/Misunderstanding of Free File Program. 


Taxpayers lack awareness that Free File exists or misunderstand the program 


Recommendations 


Findings 


Taxpayers demonstrated a lack of knowledge in the testing 
sessions and during recruitment screening about what the 
Free File program is and whether they used it previously 
(versus a free commercial product). Taxpayers also indicated 
during the interviews that they expected the IRS to directly 
provide free tax filing software rather than being taken to an 
external website. Evidence for these findings: 

Usability Issues 

• Seven taxpayers expressed surprise that Free File 
software providers are commercial companies, not 
directly offered through the IRS {Select Offer 
without Lookup Tool 4). 

• Taxpayers mistook finding an e-file provider for 
finding a Free File software offer {Locate Free File 

4)- 

Qualitative/Debriefing Support 

• Four taxpayers conflated free versions of FFA 
member software offerings with Free File during 
debriefing interviews. 

• Five taxpayers reported not having heard about Free 
File before the prior year. 

• Three taxpayers suggested the IRS advertise Free 
File more. 

Eligibility Screening 

• Of 36 taxpayers deemed eligible to participate in the 
study, 28 percent expressed uncertainty as to 
whether they used Free File or a free version of 
commercial software 

Taxpayer Pathways/Metrics 

• Four taxpayers ended their Free File search on a 
commercial tax software provider website and never 
sought IRS.gov, suggesting that they did not realize 
that Free File offers should be accessed via IRS.gov. 

• MITRE’s study design (e.g., recruitment letters and 
interview procedures) may have raised awareness 
among participants about Free File and the IRS’ 
involvement. The majority of search keywords used 
to find Free File in the interviews involved some 
combination of “IRS” and “free.” Keywords with 
this combination tended to lead taxpayers 
successfully to the Free File landing page. 


MITRE recommends conducting research on eligible 
taxpayers’ awareness of the Free File program, 
focusing on taxpayers who prepare their own taxes. 

The findings here suggest that many taxpayers do not 
know about Free File, conflate it with free commercial 
offerings, or assume it is a service offered directly by 
the IRS rather than through commercial partners. 
However, these results may be biased by the fact that 
MITRE’s recruitment materials referenced Free File; 
research that systematically evaluates taxpayer 
awareness is needed. 

MITRE recommends adding language to the Free File 
software offers page to clarify the relationship between 
the IRS and the companies who provide the Free File 
software offers. 

MITRE recommends clarifying language used on 
IRS.gov to reflect mention of the Free File program 
wherever possible (e.g., instead of a link reading “File 
Taxes for Free,” specify “File Federal Return with 
Free File program”). 

MITRE recommends using consistent program names 
in all communications to taxpayers about the Free File 
program and repeating these program names on 
IRS.gov web pages. MITRE suggests homing in on the 
phrase “IRS Free File” as a phrase that communicates 
the IRS’s involvement in the program and led to 
taxpayers successfully finding the Free File landing 
page in the usability testing sessions. 

MITRE recommends on the Free File landing page and 
on the Free File software offerings page, including a 
plain-language statement saying that the IRS 
acknowledges the software providers as providing 
legitimate Free File software offers. Describe this in a 
way that assure the site visitor that the links below are 
legitimate. 

MITRE recommends that the IRS share the FFA 
member website suggested solutions and 
recommendations within this report with the FFA. 
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Table 4. Taxpayers Understanding of EITC Eligibility Criteria. 


Taxpayers do not understand EITC eligibility criteria 

Findings 

Recommendations 

Taxpayers struggled to answer the EITC eligibility question using 
the software lookup tool. When in doubt, some taxpayers opted to 
select that they were eligible for EITC, which resulted in software 
offer results that they were not eligible for. This could cause 
problems for taxpayers once they start to complete a return and 
discover themselves to be ineligible. Taxpayers did not seem to 
understand that incorrectly completing the tool could result in the 
consequence of selecting an offer for which they were not 
eligible. 

The EITC modal on the software lookup tool and EITC assistant 
did not help resolve the issue for many taxpayers. Multiple 
participants read the EITC modal in the lookup tool and believed 
they understood the information yet selected the incorrect answer 
on the lookup tool question. All of the errors committed on the 
software lookup tool were driven by the EITC eligibility 
question. 

Usability Issues 

• Eight taxpayers selected “yes” on the EITC question in 
the lookup tool despite being unsure of answer. {Select 
Offer with Lookup Tool 2 ) 

• Five taxpayers believed that the lookup tool would verify 
their responses, especially on EITC eligibility. (Select 
Offer with Lookup Tool 1) 

• Six taxpayers misunderstood the information in the EITC 
modal popup {Select Offer with Lookup Tool 4, 5, 6, 9) 

• Two taxpayers were confused after using EITC assistant 
tool. (Select Offer with Lookup Tool 8) 

• Three taxpayers had preconceived notions regarding 
eligibility criteria and committed errors as a result. 

{Select Offer with Lookup Tool 6, 7) 

• Two did not know how to answer the EITC question and 
wanted to leave it blank or leave the lookup tool. {Select 
Offer with Lookup Tool 3) 

Taxpayer Pathways/Metrics 

• 29 percent of taxpayers committed an error on the 
software lookup tool; 100 percent of these errors were 
due to incorrect response on the EITC eligibility 
question. 

MITRE recommends making changes to the 
software lookup tool to clarify that the tool does 
not verify taxpayer responses, particularly on 

EITC eligibility. This could be done by including 
a statement in the tool description that says, 

“Please enter your information accurately. The 
tool does not verify your responses.” Or, “The tool 
does not confirm you qualify for certain credits 
but makes suggestions for software offers based 
on the information you enter. It does not guarantee 
that the Free File software offers suggested meet 
your specific needs.” 

MITRE recommends reworking the software 
lookup tool’s EITC eligibility question. Consider 
breaking out EITC eligibility into a series of 
related questions (e.g., about income, anticipated 
filing status, and number of dependents) to 
determine EITC eligibility, rather than having 
taxpayers evaluate their own eligibility. 

If the IRS stated on the Free File site that all 
taxpayers eligible for EITC are eligible for Free 
File, the question could be removed from the tool. 
MITRE recognizes this would be a topic for 
negotiation between the IRS and FFA because of 
how that could impact some companies’ 
appearance in the tool. 

If the IRS decides to continue to require taxpayers 
to evaluate their own eligibility, MITRE 
recommends linking the question to the EITC 
eligibility tool, but ONLY after conducting 
usability testing on the EITC eligibility tool and 
addressing the results. 

MITRE recommends making EITC eligibility a 
non-required response on the lookup tool or 
introducing a “don’t know” option. 
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Table 5. Usability Issues with Software Lookup Tool. 


Usability issues with the Software Lookup Tool 

Findings 

Recommendations 

The testing revealed usability issues and misunderstandings about 
the purpose of the tool that affected taxpayers’ ability to use the 
tool and answer the questions accurately. Most critically, some 
taxpayers did not understand what the tool was for, and believed 
it was confirming eligibility for Free File in general, rather than 
helping down select to a list of software offers for which they 
were eligible. Misunderstandings of the tool could lead to 
taxpayers committing errors and not realizing the consequences 
of those errors, such as using a tax software for which they are 
not eligible. Not noticing or not understanding the tool resulted in 
taxpayers who would otherwise appreciate help selecting a 
software offer not using the tool for its intended purpose. 

Usability Issues 

• Five taxpayers believed that the lookup tool would verify 
their responses to questions on the tool. (Select Offer with 
Lookup Tool 1) 

• Four taxpayers expressed the belief that the lookup tool’s 
purpose was to verify their eligibility for Free File. 

(Select Offer with Lookup Tool 11) 

• Six taxpayers misunderstood the information in the EITC 
modal popup and answered the question incorrectly as a 
result. (Select Offer with Lookup Tool 4, 5, 6 , 9) 

• Six taxpayers struggled to select no state at the state 
return drop down. (Select Offer with Lookup Tool 10) 

• Four taxpayers did not notice the lookup tool button. 
(Select Offer with Lookup Tool 12) 

Debriefing/Qualitative Support 

• Some taxpayers who did not use the lookup tool 
expressed a desire during debriefing for something that 
offered the same functionality as the lookup tool, 
suggesting that they did not notice it or understand what 
it was. “I would like to have it (IRS site) ask what state I 
live in and it provide appropriate results. Can ask three 
questions to reduce the number of options: Is your 
income less than - and click the right income level. What 
state do you live in? And is your age less than 56?” 

Taxpayer Pathways/Metrics 

• Seven of 17 taxpayers that opted to use the lookup tool 
without interviewer prompting committed errors or were 
unable to complete the task of selecting a Free File 
software offer. 

• Thirty-six percent of taxpayers opted not to use the 
lookup tool or did not notice it. 

For the software lookup tool placement: 

MITRE recommends moving the software lookup 
tool box higher on the software offers page and 
rewording the explanation text to draw users’ 
attention and make the purpose clearer. 

For the software lookup tool label and 
description: 

MITRE recommends relabeling the software 
lookup tool button and header text to clarify its 
purpose. Relabel the button “Find a Free File 
software offer” and remove the text on the lookup 
tool page that reads “Verify Eligibility.” 

MITRE recommends including a statement in the 
tool description that says, “Please enter your 
information accurately. The tool does not verify 
your responses.” Or, “The tool does not confirm 
you qualify for certain credits but makes 
suggestions for software offers based on the 
information you enter. It does not guarantee that 
the Free File software offers suggested meet your 
specific needs.” 

For alternatives and more details, see the 
suggested solutions for the issues Select Offer with 
Lookup Tool 1 and Select Offer with Lookup Tool 
11. 

For the software lookup tool state questions: 

MITRE recommends that after this statement, “If 
you need to prepare and e-file a free state return, 
please select your state, otherwise click continue.” 
include an additional statement on a new line 
saying, “[Names of states] do not have a tax on 
wages, so they do not appear on the response 
options. [New line.] If your state does have an 
income tax and does not appear in the response 
options, your state is not eligible for free filing. 

You may still be eligible to file your federal 
returns through Free File in this case.” 

For more details and alternative solutions, see the 
suggested solutions for issue Select Offer with 
Lookup Tool 10. 
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Table 6. Taxpayers Difficulty Locating Free File Landing Page through IRS.gov. 


Taxpayers have difficulty locating Free File landing page through IRS.gov channels 


Findings 


Recommendations 


Taxpayers demonstrated difficulty navigating through 
IRS.gov to find the Free File landing page. However, most 
taxpayers did eventually find the Free File landing page. An 
equal number of participants experienced sub-optimal and 
optimal routes when starting from IRS.gov. Almost all 
taxpayers started the search for Free File from a search 
engine query, suggesting that the IRS should expect that 
taxpayers will enter the Free File system on IRS.gov from 
search results or locations other than IRS.gov’s home page. 
Evidence for these findings: 

Usability Issues 

• Three taxpayers navigated to various IRS pages 
while attempting to find Free File. They thought that 
Free File was hard to find, were not sure how they 
arrived at Free File once they did, or never found it 
on their own at all. (Locate Free File 1) 

• One taxpayer went to multiple pages on the IRS 
website including the Free Fillable Forms, Find a 
Tax Professional, and the Forms and Instructions 
pages. This taxpayer did not find the Free File page 
on their own. {Locate Free File 1 ) 

• One taxpayer made it to the Free File landing page 
but then had trouble proceeding to the list of Free 
File software offers. The taxpayer veered away from 
the Free File landing page and navigated to the 
Filing for Individuals page. {Locate Free File 2) 

• One taxpayer navigated to the Authorized e-file 
Provider page. This taxpayer initially found the Free 
File landing page but then navigated to the page with 
authorized e-file providers. {Locate Free File 4) 

Taxpayer Pathways/Metrics 

• Twenty-four participants were able to find the Free 
File landing page. 

• Of the 24 participants that ultimately found the Free 
File landing page independently, 10 followed 
suboptimal paths, meaning that they were frustrated 
or confused along the way and visited other, 
unnecessary pages in the course of their search. 

• Twenty-six taxpayers started their search for Free 
File with an internet search engine query (e.g., “Free 
tax filing”). 

o The search terms that most frequently led to 
the Free File landing page included both 
“IRS” and “Free File.” 


MITRE recommends clarifying language used on 
IRS.gov to mention of the Free File program wherever 
possible (e.g., instead of a link reading “File Taxes for 
Free,” specify “Do Your Federal Taxes for Free with 
Free File”). 

MITRE recommends visually prioritizing the button or 
link to the Free File program on the IRS homepage by 
moving it higher on the list. 

MITRE recommends that the IRS take steps to 
accommodate the fact that most taxpayers will enter 
the Free File system via an internet browser search, 
rather than through IRS.gov directly. Advertising that 
uses the phrase “IRS Free File” may prime taxpayers 
to search on that phrase, leading to the Free File 
landing page. MITRE recommends promoting a single 
program name like “IRS Free File” to help taxpayers 
remember this phrase, use it for internet searches, and 
scan for it on IRS.gov. 

Standardize the terms used across IRS.gov and any 
communications about the Free File program. Avoid 
mixing terms and headers (e.g., using “File Taxes for 
Free” on one web page and “Free File” on another). 
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Table 7. Overwhelming Amount of Free File Vendor Options and Information with Offers. 


Overwhelming number of Free File Software offers, and information provided with the offers 


Findings 


Recommendations 


Taxpayers were overwhelmed by the Free File software 
options and by the information provided along with options 
on IRS.gov Free File-related pages (e.g., Free File landing 
page and Free File software offers page). They were 
overwhelmed by the amount of information they saw when 
they reached the Free File software offers page, including the 
paragraphs above the Free File software lookup tool box and 
the Free File software offers list below the lookup tool box. 
Taxpayers also found that the lookup tool returned too many 
options. Taxpayers found it difficult to select an offer due to 
the number of choices and amount of information presented 
to them. Evidence for these findings: 

Usability Issues 

• Five taxpayers were overwhelmed by the amount of 
information on the Free File software offers page, 
particularly the Free File software offers. {Select 
Offer without Lookup Tool 2) 

• Three taxpayers perceived too many options returned 
from the lookup tool and had trouble choosing an 
offer. {Select Offer with Lookup Tool 13) 

• Five taxpayers did not understand if they needed to 
meet all or only some of the eligibility criteria on the 
Free File software offers list page in order to qualify. 
They did not understand if the criteria were “and” or 
“or” statements. This confusion resulted in taxpayers 
not knowing which Free File software offers they 
were eligible for. Some understood Free File 
software offers as being ONLY for military or EITC. 
{Select Offer without Lookup Tool 1) 

Qualitative/Debriefing Support 

• Five participants felt the biggest difficulty was that 
there were too many options to choose from on the 
IRS website 

o “There are too many choices. If there are ten 
there and four qualify, then it’s like eenie 
meenie miney mo.” 


MITRE recommends adding language to the Free File 
software offers page to clarify the relationship between 
the IRS and the companies who provide the Free File 
software offers. 

MITRE recommends offering visual relief to break up 
the visual monotony of a litany of text boxes. Visuals 
like the logos of the offerings, inserted into each box 
or next to the appropriate Free File software offer 
name, can help taxpayers quickly differentiate between 
offerings and remember software providers they wish 
to scroll back to or revisit. Logos also help clarify that 
the offerings are legitimate but are not IRS or 
government affiliates, which addresses taxpayers’ 
confusion that Free File software is provided directly 
by the IRS. 

MITRE recommends clearly separating between each 
of the groups by criteria. Group all of the “and” criteria 
together, and clearly separate all of the “or” criteria. 

MITRE recommends creating a table that uses plain 
language and compares the eligibility criteria of each 
offer, rather than presenting many individual boxes on 
the software offers page. Present information 
consistently across the offerings’ descriptions. 
Alternatively, MITRE recommends including a tooltip 
with each of the resulting offers from the lookup tool 
that when clicked, allows taxpayers to see the 
eligibility criteria associated with each offer. This 
would give taxpayers the option to revisit the offers’ 
eligibility criteria without putting too much 
information on the results page and without requiring 
taxpayers to leave the page. 
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Table 8. Taxpayers Experience Selecting Software and Strategies for Selection. 


Taxpayers’ experience selecting a software offer: Strategies used to select offers 


Findings 


Recommendations 


The study collected a wealth of information about how 
taxpayers navigate the process of choosing a Free File 
software offer. Interestingly, the majority of taxpayers 
reported reasons other than eligibility criteria for selecting 
Free File software offers, in part because the eligibility 
criteria were long and complex. Taxpayers rely on 
familiarity with company names, particularly the large 
software providers, and previous experience with tax 
software. When in doubt, taxpayers drew on heuristic 
devices to help themselves establish the legitimacy of 
software providers, such as keywords in software providers’ 
names. In part, these strategies were necessitated by the 
sense that there were too many options and it was difficult to 
distinguish between them. Some taxpayers mistook 
commercial providers with similar keywords as legitimate 
IRS Free File program selections. 

Usability Issues 

• Taxpayers were overwhelmed by the number of 
options, both on the software offers page and in the 
lookup tool results. ( Select Offer without Lookup 
Tool 2 and Select Offer with Lookup Tool 13) 

• Taxpayers expressed a desire to see the eligibility 
requirements repeated on the lookup tool results list, 
to help them choose from the list of Free File 
software offers. 

Qualitative/Debriefing Support 

• The majority of taxpayers cited reasons other than 
eligibility criteria for choosing a Free File software 
offer. Prior experience and name recognition were 
the top two reasons, with a total of 45 mentions. 

• Eligibility criteria were mentioned as the reason for 
selection 13 times. 

Taxpayer Pathways/Metrics 

• Nine taxpayers navigated to a commercial website as 
part of the process while they were searching for 
Free File. Four of the nine ultimately stayed on a 
commercial page, not realizing that they had not 
selected a Free File offer. 


MITRE recommends that the IRS take steps on its Free 
File web pages to help make eligibility clearer and less 
overwhelming to taxpayers, in order to maximize 
taxpayers’ ability to select an offer based on eligibility , 
not criteria such as name recognition, keywords used 
in names, or picking at random. 

To help taxpayers focus on eligibility criteria: 

MITRE recommends creating a table that uses plain 
language and compares the eligibility criteria of each 
offer in a grid format. Present information consistently 
across the offerings’ descriptions. 

MITRE recommends including a tooltip with each of 
the resulting offers from the lookup tool that when 
clicked, allows taxpayers to see the eligibility criteria 
associated with each offer. This would give taxpayers 
the option to revisit the offers’ eligibility criteria 
without putting too much information on the results 
page and without requiring taxpayers to leave the page. 

MITRE recommends incorporating the term “IRS Free 
File” wherever possible to help distinguish the IRS- 
associated partnership program from commercial sites 
that mirror the language and look and feel of Free File 
programs. 
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Table 9. Difficulty Reporting Adjustments to Income, Choosing Filing Status, and Navigating 

Systems. 


Difficulty reporting adjustments to income, choosing filing status, and navigating systems 


Findings 


Recommendations 


General usability issues across multiple software providers 
included finding where to enter student loan interest 
information, selecting a filing status, and navigating through 
the filing systems. Taxpayers were not sure where to enter 
student loan information and often found it where they were 
not expecting. Taxpayers had a hard time deciding between 
selecting head of household or single. They did not 
understand the flow, or sequence of pages and information, 
of various aspects of the tax preparation systems. Evidence 
for these findings: 

Usability Issues 

• Seven taxpayers were not sure where to enter their 
student loan information from Form 1098-E while 
filing a return using two out of the three Free File 
software offers. {File a Return 1 ) 

• Six taxpayers were uncertain or confused about 
either selecting head of household or single as their 
filing status. Taxpayers expected more information 
within the system to help them make the correct 
choice. Instead, they had to turn to an external 
source for more information. {File a Return 2, File a 
Return 12) 

o If they chose head of household, some 
taxpayers expected to see additional 
questions or prompts to confirm their 
selection but did not receive them. {File a 
Return 2) 

o System prompts asking taxpayers if they 
wanted to change their filing status were 
confusing and undermined taxpayer 
confidence {File a Return 12) 

• Three taxpayers did not understand the path or flow 
of information presented to them while filing. They 
lost track of where they were in the filing process, 
had trouble navigating menus, or did not see the 
pages they were expecting to see. {File a Return 11) 

• Three taxpayers were confused about what to do 
once they chose the quick file option. They were not 
familiar enough with tax preparation to enter each of 
the relevant forms they had for filing. {File a Return 

3 ) 


For adjustments to income: 

MITRE recommends including an education expenses 
section with a checkbox for student loan payments on 
the page where taxpayers are selecting the items of 
basic information such as income and dependents that 
apply to them. Checking the box for student loan 
payment should trigger the presentation of a student 
loan interest payment deduction page when taxpayers 
later enter their information for deductions. 

MITRE recommends making sure navigation is usable 
and descriptive to ensure users can go back and find 
where to enter student loan information if they miss it 
on their first pass. 

For choosing a filing status: 

MITRE recommends providing step-by-step prompts 
to help taxpayers make the correct filing status 
selection, particularly between single and head of 
household. 

MITRE recommends that in the filing status prompt 
alert, explain in more detail why a suggestion was 
made (i.e., explain what qualifies someone for head of 
household), and affirm that the participant does not 
have to change their selection. 

Alternately, MITRE recommends that when a taxpayer 
uses the filing wizard to determine their status, do not 
give them any prompts to make changes to their filing 
status unless they later make selections or enter 
information that contradicts what they have entered 
into the filing wizard. 

For flow of information: 

MITRE recommends using symbols and color 
saturation in areas such as the side navigation menu to 
indicate where in the process the taxpayer is and the 
status of the section. For example, use check marks to 
indicate that the taxpayer has completed a section and 
an empty circle to show incomplete sections, and use 
color or color saturation differences to highlight where 
in the menu the taxpayer currently is. 
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Table 10. Taxpayer Satisfaction and Ease of Use Ratings for the Free File System. 


Taxpayer satisfaction and ease of use ratings for the Free File system 


Findings 


Recommendations 


The study used two questionnaires to collect self-report 
metrics on participants’ perceived ease of use of the three 
usability tasks, and overall perceived satisfaction regarding 
the Free File system. The findings show that overall, 
taxpayers rated the usability of the Free File system a “B.” 
The filing task was the most difficult for participants, which 
is not surprising given that completing a tax return is a more 
complex task than browsing the internet or navigating 
IRS.gov. Usability scores varied by Free File software offer. 
There was some evidence that usability scores varied by age 
and education level, though these results should be 
interpreted with caution given the small sample size. 

Questionnaire Scores/Analysis 

• Participants rated task 3 as harder than tasks 1 and 2; 
in line with task 3 being inherently more complex 
than tasks 1 and 2. 

• Participants’ ratings of the Free File system overall 
varied by which Free File software offer they used to 
file a return, suggesting that the software experience 
colored taxpayers’ perceptions of the entire 
experience, including IRS site interactions. 

• Patterns in usability ratings by education were 
unclear: 

o High school-equivalency group had lowest 
per-task ease for Tasks 1 and 2 (SEQ) but 
highest overall usability (UMUX) 

• Youngest age tercile (20-35 years) reported the best 
usability across entire experience, but there was not 
a conclusive age group trend. 


MITRE recommends performing further research to 
explore these findings, including usability testing and 
iterative design processes. Iterative usability testing 
should be conducted with various demographic groups, 
using larger sample sizes than those in this study and 
thus gleaning more reliable results. Usability testing 
that includes new filers is recommended. 

Vendor usability was associated with overall usability 
ratings of the Free File system as a whole. To increase 
taxpayer satisfaction with Free File, the IRS can 
encourage or require Free File software providers to 
ensure their Free File offerings are usable by 
conducting testing or adhering to best practices. The 
IRS can also communicate more clearly to taxpayers 
that Free File software offers are non-IRS and 
encourage taxpayers to explore multiple options. This 
could increase the likelihood that taxpayers will return 
to the Free File program if they have a bad experience 
with a software offer, rather than assume that the entire 
program is equally difficult to use. 
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4 Usability Findings 

In this section, MITRE describes the usability issues identified during the taxpayer 
testing sessions. Taxpayers performed three main tasks during the testing sessions: 
locate the Free File landing page, select a Free File software offer using fictitious 
information, and file a return using fictitious information. 

The focus of this section is on the elements of web design, wording, or organization 
that caused taxpayers to commit errors, express frustration, or be unable to complete 
any of the tasks. Suggested solutions are provided to address usability issues on an a 
la carte basis. 

MITRE identified 37 distinct usability issues. Some major issues included: 

• Difficulty navigating from IRS.gov to the Free File landing page 

• Misunderstanding EITC eligibility while using the lookup tool 

• Difficulty entering adjustments to income, particularly student loan interest, 
while filing 

4.1 Demographic Characteristics of Testing Participants 

Table 11 shows the demographic breakdown of all taxpayers that completed the interview. The 
data in Table 11 were reported by the participants during phone screenings or interview sessions, 
not derived from tax return data in CDW. At a glance, the vast majority of participants filed as 
single (90 percent). Median age was 46 years (interquartile range 34-54 years). Compared with 
free filers nationally, 10 the final interview sample had a higher percentage of single filers and 
skewed older than free filers nationally (see Table 77 for national free filer characteristics). Of 
the interview sample, 45 percent reported a household income between $15,001 and $35,000 and 
the most co mm on educational level was a bachelor’s degree (44 percent) followed by having a 
master’s degree (31 percent). Note that participant recruitment ultimately depended upon the 
taxpayers themselves contacting MITRE after receiving an invitation letter; therefore, MITRE 
was not able to dictate the characteristics of the taxpayers who participated beyond the 
recruitment plan discussed in Section 2.2. 


10 In TY 2018, the full population of U.S. free filers had median age 32 years, mean age 37.9 years, and 71 percent 
filed as single. 
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Table 11. Demographic Characteristics of Participants. 



Participant 

Count 

Percent of 
Total 

Total Number of Participants 

29 



Filing Status 

Single 

26 

90% 

Married Filing Joint 

0 

0% 

Married Filing Separate 

0 

0% 

Head of Household 

3 

10% 

W ido w/W ido wer 

0 

0% 

Age (Quartiles) 

20-33 

8 

28% 

34-45 

7 

24% 

46-53 

7 

24% 

54-85 

7 

24% 

Household Income 

Less than $15,000 

6 

21% 

Between $15,001 and $35,000 

13 

45% 

Between $35,001 and $50,000 

6 

21% 

Between $50,001 and $75,000 

3 

10% 

Between $75,001 and $90,000 

1 

3% 

More than $90,000 a year 

0 

0% 

Unknown 

0 

0% 

Highest Level of Education 

No schooling completed 

0 

0% 

Some school but did not complete high school 

0 

0% 

Received high school diploma, GED or alternative 

5 

17% 

Associate’s degree 

1 

3% 

Bachelor’s degree 

13 

45% 

Master’s degree 

9 

31% 

Professional degree beyond bachelor’s degree 

0 

0% 

Doctorate degree 

1 

3% 

Unknown 

0 

0% 

Military Pay 

Participant and/or spouse received military pay 

0 

0% 

Zip Code 

60613 

2 

7% 

60618 

4 

14% 

60625 

4 

14% 

60640 

11 

38% 

60660 

3 

10% 

60657 

5 

17% 
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4.2 Usability Issues with Suggested Solutions 

This section describes issues, problems, points of confusion, and causes of errors and 
misreporting that came up during the usability scenarios/testing that MITRE interviewers 
administered to participants. Participants’ opinions—if not related to a cause of errors, problems, 
or frustration while performing the task—are not included in this section. 

Also, in this section are issue-level suggested solutions for fixing specific usability issues. Note 
that these suggestions are meant to be a la carte, as some may conflict with others. MITRE 
provided suggested solutions to each taxpayer issue in Task 3 - File a Return Using a Pre¬ 
assigned Vendor. These solutions pertain to specific Free File software offers. The IRS will need 
to determine what works best for them in the full context of the IRS Free File program, including 
whether to share suggested solutions with the Free File software providers MITRE selected 
randomly to test. 

All usability issues were given an ID for easy referencing that is comprised of the task and a 
number (e.g., “Locate Free File 1,” “Select Offer without Lookup Tool 1,” “Select Offer with 
Lookup Tool 1,” “File a Return 1”). 

For more information on the usability issues analytical approach see Section A4.1 Usability 
Data. For a full list of all usability issues with no accompanying narrative text, see Appendix B. 

4.2.1 Reporting Usability Issues: Presentation of Issues Example 

The MITRE team used summary tables (Table 12) to present each of the usability issues. 
Following each table is a full description of the issue and suggestions to resolve that issue. 


Table 12. Example Usability Issue Table. 


Task 

Website 

Page/Section 

Issue 

Severity 

Count 

Example 

task 

Example 

website 

Example 
page/section 
of website 

Example issue description 

Low, 

medium, 

high 

# of 

participants 


4.2.2 Reporting Usability Issues: Severity Ratings 

The MITRE team classifies usability issues by severity rating. These ratings allow a quick 
reference to gauge how serious an issue was—was it a showstopper for some taxpayers, or 
merely a point of confusion? Did errors result for some taxpayers who encountered this problem? 
Note that severity ratings do not necessarily tie to the frequency of the issues among the 
participants and are treated as distinct concepts [2], 11 

1 . High severity issues. Issues that caused users to commit major errors or stopped the 
taxpayer from completing the task at hand. For example, entering an incorrect amount 
that causes cascading errors throughout the tax return and to the final refund amount. 


11 Severity is not tied to frequency because regardless of how many people experience a high severity issue, it is 
problematic if anything about the website design causes taxpayers to be unable to complete the task or complete it 
without any major errors. 
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2. Medium severity issues. Issues that caused major frustration that the participant was 
able to eventually resolve on his/her own, or issues that caused minor errors. For 
example, not ticking a required checkbox but otherwise recording other amounts 
correctly. 

3. Low severity issues. Issues that caused some frustration or caused the user to express 
dissatisfaction with the experience of completing the form, but the responses were correct 
in the end. 

4.2.3 Taxpayer Usability Issues While Performing Task 1 - Locating the Free File 
Landing Page 

In this task, the interviewers instructed taxpayers to imagine that they learned there was a 
program available through the IRS that allowed them to file their federal tax return for free and 
to use an internet browser of their choice to locate that program. Below are the issues taxpayers 
encountered while performing this task. 

4.2.3.1 Locate Free File Landing Page 1 


Table 13. Locate Free File Landing Page 1. 


Task 

Website 

Page/Section 

Issue 

Severity 

Count 

Locate 

Free File 
Landing 
Page 

IRS.gov 

IRS 

Homepage 

Taxpayers had trouble 
navigating from the IRS 
homepage to Free File page 

High 

3 


Description of the Issue 

Once they were on the IRS homepage, taxpayers had difficulty finding the Free File landing 
page, and one taxpayer was not able to find the page without assistance. Taxpayers navigated to 
various IRS pages while attempting to find Free File. They thought that Free File was hard to 
find, were not sure how they arrived at Free File once they did, or never found it on their. 

One taxpayer got tied up in the Interactive Tax Assistant. This taxpayer eventually found the 
Free File page but was not sure how they got there, saying, “This is very confusing, I’m not sure 
how I got here.” 

Another taxpayer went to multiple pages on the IRS website including the free tillable forms, 
find a tax professional, and the forms and instructions pages. This taxpayer did not find the Free 
File page on their own. 

Though it took them several tries to find it, another taxpayer found the link to the Free File page 
in the righthand blue box under the File drop-down menu. This taxpayer said, "it was in the 
middle. Free File was hiding." 

This issue is related to Locate Free File 2. 

Suggested Solutions 

On the IRS homepage, change the label “File Your Taxes for Free” to “Do Your Federal Taxes 
for Free with Free File.” This change may be help taxpayers find the Free File page as it would 
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both include the term “Free File” and relate better to the page that opens when you click the “File 
Your Taxes for Free” button. The task-oriented language better aligns with how the user 
conceptualizes their goals on the page. 

Visually prioritize the button or link to the Free File program by moving it higher on the list. 
Other approaches could be a different shape or larger size of button, or a button that is visually 
called out with color. A unique and more relevant icon could also be used; the current folder icon 
is shared with “View Your Account,” and while it relates linguistically to the concept of filing, it 
does not explicitly relate to the ability to do tax returns for free. 

Arrange the linking elements on the page in order of priority. 


aiRs 


Pay 


-H*' 



^ Get Your Refund Status 

A 

D View Your Account 


Q Where’s Your Amended Return? 
Get Answers to Your Tax Questions 


Help | News ! Language v | Charities & Nonprofits Tax Pros 


Credits & Deductions 


Forms & Instructions 


[=1 Get Your Tax Record 
Make a Payment 
© Visit the Trucking Center 
File Your Taxes for Free 



Figure 3. IRS Home Page (IRS.gov). 


4.2.3.2 Locate Free File Landing Page 2 


Table 14. Locate the Free File Landing Page 2. 


Task 

Website 

Page/Section 

Issue 

Severity 

Count 

Locate 

Free File 
Landing 
Page 

IRS.gov 

Free File 
Landing 
Page 

Taxpayer had difficulty 
navigating Free File landing 
page (on IRS.gov) 

Medium 

1 


Description of the Issue 

A taxpayer made it to the Free File landing page but then had trouble proceeding to the list of 
Free File software offers. The taxpayer veered away from the Free File landing page. They 
navigated from the Free File landing page to the filing for individuals page by clicking 
"individuals" on the top-level menu. The taxpayer read the “what you need to know” section then 
selected the Form 1040 link, and then selected the current year link. At this point, the interviewer 
intervened to redirect the taxpayer and help move the task forward. As Figure 4 and Figure 5 
(laptop views) illustrate, the button to move to the offerings page is located below the fold on 
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page loading and is thus easy to overlook. The offerings page button is also visually identical 
(text asides) to the “start tillable forms now” button, which means the user is not guided toward 
free e-filing. Taxpayers who are new to e-filing or unaware of the Free File option may go 
straight to the tillable forms option. The text describing the income level distinction between the 
two options is not visually striking and site visitors may gloss over it. 

This issue is related to Locate Free File 1. 

Suggested Solutions 

Bring the Free File offerings button higher up the page so that it is visible without the user 
scrolling down. Removing the alert box at the top of the content area may achieve this. 

The content in the alert box seems less important than the filing options. Remove the alert box so 
that the site visitor’s gaze is drawn toward the filing options rather than the alert. 

Show the income below/above cutoff descriptions side by side rather than on top of each other to 
contrast them more clearly as distinct options. The current layout may be interpreted at a glance 
as being a list rather than contrasting options. 

Keep link and button text task-oriented. This means that the “Start Free File Now” button would 
be easier to understand if it read something akin to “Find Software to E-File for Free.” Taxpayers 
who do not know much about the Free File program may be confused by Free File presented as a 
task. 


28 

PRE-DECISIONAL - FOR INTERNAL IRS W&l USE ONLY - NOT FOR PUBLIC RELEASE 



PRE-DECISIONAL - FOR INTERNAL IRS W&l USE ONLY - NOT FOR PUBLIC RELEASE 



File Pay Refunds Credits & Deductions Forms & Instructions 

Home > File > Individuals > How to File > Free File: Do Your Federal Taxes for Free 


Free File: Do Your Federal Taxes for Free 


Individuals 

How to File 

Determine Filing Status 

Free File 

e-File Options 

Figure 4. Free File Program Landing Page, Above the Fold. 


English | Es panol | | | P vcckhh | TiSn eViet 

0 Free File is open for business! 

Click the Start Free File Now button to file your FREE federal return. Also available in Es panol! 

The new tax law may affect your tax return. R 


Where to File 

Your Information 

Students 

Employees 

Parents 

Military 

Seniors & Retirees 

International Taxpayers 
Businesses and Self-Employed 
Government Entities 


Choose a Free File option based on your income: 


Income below $66,000 
Free File Software 

• File your taxes with free, easy to use software 

• Free state return options are available 

• Use the Free File Software Lookup Tool to find free federal and free 
state return options 


Start Free File Now 


Free File Software Topics 


• what You Need to Get Started 

• How to Validate Your Return 


Income above $66,000 
Free File Fillable Forms 

• Must know how to do your taxes yourself 

• Does math; offers only basic guidance 

• You must have your 2017 tax return 

• State tax prep is not available 


Start Fillable Forms Now 


Free File Fillable Forms Topics 

• Available Forms 

• Military & Forei g n Addresses 

• user’s Guide 

• ttelc 


Figure 5. Free File Program Landing Page, Halfway Down the Page. 
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4.2.3.3 Locate Free File Landing Page 3 

Table 15. Locate the Free File Landing Page 3. 


Task 

Website 

Page/Section 

Issue 

Severity Count 

Locate 

Free File 
Landing 
Page 

IRS.gov 

Free File 
Landing 
Page 

Taxpayer did not notice the 
“Start Free File Now” button 

Medium 1 


Description of the Issue 

A taxpayer did not like that they needed to scroll before getting where they need to go (the “Start 
Free File Now” button). Scrolling caused the taxpayers to initially miss the button. 

This taxpayer said, “IRS is very wordy, and I wish the Free File banner was more displayed. I 
wasn’t sure this was the button. The banner that was up top was not the navigation button- 
looked like a button to me but I had to go further down the page. I design web pages, so I have 
some experience.... People don’t have patience anymore. Scrolling down is like a task. Put it 
front and center and make it user friendly.” 

Figure 6 shows the current Free File landing page (https://www.irs.gov/filing/free-file-do-your- 
federal-taxes-for-free). Taxpayers must scroll down to see the rest of the page and the “Start Free 
File Now” button shown in Figure 7. 



File Pay Refunds Credits & Deductions Forms & Instructions 

Home > Eite > Individuals > How to File > Free File: Do Your Federal Taxes for Free 

Free File: Do Your Federal Taxes for Free 


English | Es panol | | | P vcckhh | TieneViet 


Individuals 

How to File 

Determine Filing Status 
Free File 
e-File Options 
Mailing Addresses 
When to File 


^ Free File is open for business! 

Click the Start Free File Now button to file your FREE federal return. Also available in Es panol! 
The new tax law may affect your tax return. R 


Welcome to Free File, where you can prepare and file your federal individual income tax return for free using tax-preparation- 
and-filing software. 

Let Free File do the hard work for you with brand-name software or Free File Fillable Forms. 


Figure 6. Free File Landing Page. 
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Mailing Addresses Welcome to Free File, where you can prepare and file your federal individual income tax return for free using tax-preparation- 

and-filing software. 

>n to File Let Free File do the hard work for you with brand-name software or Free File Fillable Forms. 


Where to File 
Your Information 
Students 


Choose a Free File option based on your income: 

Income below $66,000 
Free File Software 


Employees 

Parents 

Military 

Seniors & Retirees 


• File your taxes with free, easy to use software 

• Free state return options are available 


o find free federal and free 


state return options 


Start Free File Now 


Free File Software Topics 


• What You Need to Get Started 

• How to Validate Your Return 

• About the Free Fjje program 


International Taxpayers 


Businesses and Self-Employed 


Government Entities 


Income above $66,000 
Free File Fillable Forms 

• Must know how to do your taxes yourself 

• Does math; offers only basic guidance 

• You must have your 2017 tax return 

• State tax prep is not available 


Start Fillable Forms Now 


Free File Fillable Forms Topics 

• Available Forms 

• Minimum Computer Requirements 

• Military & Forei g n Addresses 

• User's Guide 

• Help 


Figure 7. Free File Landing Page—Start Free File Now. 


Suggested Solutions 

Solution suggestions are similar to those in Locate Free File 2. Remove the blue information box 
in order to move up the “Start Free File Now” button and make it visible on the page without 
having to scroll down. Provide the two income level options (Free File and fillable forms) as 
options users can scan side by side, which will also reduce the amount of scrolling required. 


4.2.3.4 Locate Free File Landing Page 4 


Table 16. Locate the Free File Landing Page 4. 


Task 

Website Page/Section 

Issue 

Severity 

Count 

Locate 

Free File 
Landing 
Page 

Free File 
IRS.gov Landing 

Page 

Taxpayer confused between 
finding a Free File provider 
and an e-file provider 

Low 

1 


Description of the Issue 

A taxpayer was supposed to find a Free File software offer but ended up on the authorized e-file 
provider page. This taxpayer initially found the Free File landing page but then navigated to the 
page with authorized e-file providers. Upon seeing the list of individual people’s names, the 
taxpayer returned to the Free File landing page and found the “Start Free File Now” button. 

Suggested Solution 

None—The taxpayer navigated away from the Free File landing page without explanation but 
returned on their own after exploring other pages. They were able to return to the Free File 
landing page without issue. See recommendations for related issue “Locate Free File 1. ” 
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4.2.4 Taxpayer Usability Issues While Performing Task 2a - Selecting a Free File 
Software Offer (Without the Lookup Tool) 

In this task, the interviewers instructed taxpayers to use a fictitious scenario to select a Free File 
software offer starting from IRS.gov. Taxpayers did not receive any specific instruction to use 
the lookup tool, but they were free to use it on their own if they chose to do so. 

4.2.4.1 Select Offer Without Lookup Tool 1 


Table 17. Select an Offer Without the Lookup Tool 1. 


Task 

Website 

Page/Section 

Issue 

Severity 

Count 

Select an 
Offer 


Free File 

Taxpayers did not 
understand whether it was 



without the 
Lookup 

Tool 

IRS.gov 

Software 

Offers 

necessary to meet all 
eligibility criteria ("and" or 
"or") 

High 

5 


Description of the Issue 

Taxpayers did not understand if they needed to meet the all eligibility criteria on the Free File 
software offers list page in order to qualify, or only one or some of the requirements. They did 
not understand if the criteria were “and” or “or” statements. This confusion resulted in taxpayers 
not knowing which Free File software offers they were eligible for. 

One taxpayer said, “Does she have to fit all of these [bullet points], or just some of these? I tend 
to think you wouldn’t have to be, but I don’t know. It says ‘if,’ so I don’t think all of them have 
to apply, but I really don’t know.” 

Another taxpayer was specifically confused by the military criterion for one of the Free File 
software offers. They said, “Meets first two bullets, but then the third talks about military—so 
does this mean it’s just good for military or military and everything else as well...Seems like all 
the bullet points read that way.” 

Suggested Solution 

Structure all of the Free File software offers boxes like the box for 1040NOW.NET in Figure 8 
where there is a clear separation between each of the groups of criteria. All of the “and” criteria 
are grouped together and all of the “or” criteria are clearly separated. 
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1040NQW.NET 

Receive a free Federal return if: 

• If your Adjusted Gross Income (AGI) is $66,000 or less, AND 

• You live in one of the following states: AL, AR, AZ, CA, GA, IA, ID, IN, 

KY, Ml, MN, MO, MS, NC, ND, NY, OK, OR, Rl, SC, VA, VT or WV 

OR 

• Your AGI is $66,000 or less, AND you are 60 or younger, AND 

• You live in one of the following states: CO, CT, DC, DE, HI, IL, KS, LA, 

MA, MD, ME, MT, NE, NJ, NM, OH, PA, UT or Wl 

OR 

• Free for Active Military for Adjusted Gross Income of $66,000 or less 

• Available for foreign address filers 

Figure 8. Current 1040NOW.NET Free File Software Offer Information Box. 


4.2,4.2 Select Offer Without Lookup Tool 2 


Table 18. Select an Offer Without the Lookup Tool 2. 


Task 

Website 

Page/Section 

Issue 

Severity Count 

Select an 
Offer 

without the 

IRS.gov 

Free File 
Software 

Taxpayers perceived too 
many options and were 
overwhelmed by information 

Medium 5 

Lookup 

Tool 


Offers 

on the Free File software 
offers page 



Description of the Issue 

Taxpayers were overwhelmed by the amount of information on the Free File software offers 
page (https://apps.irs.gov/app/freeFile/), especially the Free File software offers list. 

One taxpayer said, “Too many options. When they list states or excluded states, I would prefer to 
just skip it... [I] would probably select the first one listed because it would take too long to 
explore all of them.” 

Additionally, the amount of information on this page did not facilitate taxpayers selecting a Free 
File software offer. Participants struggled to compare the eligibility criteria across the offers. 
While looking through the Free File software offers boxes, another taxpayer said, “It says your 
income is 66K, this other one says 35K. It’s complicated. You have to check your income and 
age. This one applies to what looks like 30 states. Others don’t mention state. To me this is a 
headache...” 

One taxpayer specifically chose to use the lookup tool because they did not understand all of the 
information below the “Free File software lookup tool” box. 

Suggested Solutions 

Create a table that uses plain language and compares the eligibility criteria of each offer. 
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Present information consistently across the offering descriptions. In each description, give 
information for the same fields in the same order. If a field does not apply, such as when a 
software provider does not limit by state or does not offer free filing for state returns, note that it 
does not apply and explain why. For example, by saying “[Field]: N/A. [Vendor] does not offer 
free e-filing for state returns.” Consistent presentation will help the taxpayer quickly and easily 
compare information between Free File software offers, as well as feel more confident in their 
filing journey. Explanations for unexpected information helps the user progress smoothly 
through their task at hand without worrying they have missed or mistaken something and thus 
performing extra checking actions. This kind of taxpayer confidence-building also works toward 
ensuring the user has as smooth an experience as possible and is more likely to return to e-filing 
in the future, rather than be scared away by a confusing or intimidating experience. 

Use plain language to describe all eligibility requirements. Use plain language in any other 
content the user needs to gain information and complete their task. Aim to write at a co mm on 
low reading level (e.g., 8 th grade reading level) to ensure that all eligible taxpayers can 
effectively access the Free File program regardless of background. 

Offer visual relief to break up the visual monotony of a litany of text boxes. Visuals like logos of 
the offerings, inserted into each box or next to the appropriate Free File software offer name, can 
help taxpayers quickly differentiate between offerings and remember software providers they 
wish to scroll back to or revisit. Logos also help clarify that the offerings are not IRS or 
government affiliates, which addresses the point of confusion in Select Offer without Lookup 
Tool 4. 

Make sure Free File software offer names on both the IRS site and the software provider site 
itself match in order to ensure users understand the connection between the Free File program 
and the software provider site. 

4.2.4.3 Select Offer Without Lookup Tool 3 


Table 19. Select an Offer Without the Lookup Tool 3. 


Task 

Website 

Page/Section 

Issue 

Severity 

Count 

Select an 
Offer 


Free File 

Taxpayers were confused by 
inconsistent references to 



without the 
Lookup 

Tool 

IRS.gov 

Software 

Offers 

states regarding eligibility on 
the Free File software offers 
page 

Medium 

5 


Description of the Issue 

Taxpayers do not understand why some Free File software offers list states and others do not. 
They also do not understand why some states such as Texas are not listed anywhere. One 
taxpayer would stop entirely if they did not see their state listed. Other taxpayers were confused 
and did not understand why the state they were looking for was not listed. 

Suggested Solutions 

Include clear, understandable information that explains the differences. Include a statement that 
explains that not all states require filing a state tax return. Include a statement that explains that a 
free state tax return may not be available for all states that require filing a state tax return. 
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Present this information about state eligibilities both in the about information at the top of the 
page and in each offering description where state does not apply. Present it consistently across 
the offering descriptions. 

This issue is related to Select Offer Without Lookup Tool 2. 

4.2.4.4 Select Offer Without Lookup Tool 4 


Table 20. Select an Offer Without the Lookup Tool 4. 


Task 

Website 

Page/Section 

Issue 

Severity 

Count 

Select an 
Offer 

without the 

IRS.gov 

Free File 
Software 

Taxpayers expected to see an 
IRS.gov-related software 

Low 

7 

Lookup 

Tool 


Offers 

offer 




Description of the Issue 

Taxpayers expected to see an offer that was directly associated with the IRS and were confused 
when they did not see one. They assumed that Free File software was provided directly by the 
IRS and not by tax preparer software providers. 

Some taxpayers commented on the amount of Free File software offers available and that they 
were looking for one from the IRS. One taxpayer said, “There are so many listed here my 
goodness. Why do they have to have so many vendors? You would think there would just be one 
for the IRS here. I want to know I am dealing directly with the IRS. I want something that looks 
familiar to me.” 

Some taxpayers were looking for more explicit language that would let them know the Free File 
software offers would not be provided by the IRS itself and the offers listed were fine to use or 
legitimate. One taxpayer said, “I wish it could have said something like ‘all of these programs 
are the programs, and that IRS itself is not the option, that it is okay to use these second party 
groups to file.” 


Suggested Solutions 

On the Free File landing page and on the offerings page, include a plain-language statement 
saying that the IRS acknowledges the software providers in question as providing legitimate Free 
File software offerings. Describe this in a way that assure the site visitor that the links below are 
legitimate. 

Clearly state that the links will go to external sites/software provider pages. 

Consider including an “open in new window” icon beside each offering link: B. This 
differentiates them from links leading to other IRS websites. 

Include software provider logos to increase the look of legitimacy. 

Ensuring taxpayers have a clear understanding of the Free File program and its boundaries also 
mitigates this issue. 
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4.2.4.5 Select Offer Without Lookup Tool 5 


Table 21. Select an Offer Without the Lookup Tool 5. 


Task 

Website 

Page/Section 

Issue 

Severity 

Count 

Select an 
Offer 

without the 

IRS.gov 

Free File 
Software 

Taxpayers misunderstood 
the order of Free File 

Low 

2 

Lookup 

Tool 


Offers 

software offers displayed 




Description of the Issue 

Taxpayers expressed the belief that the order of Free File software offers is by some criteria 
other than random. They were not sure how to assess and interpret the order of the options. 

One taxpayer believed the most popular options were placed at the top. Another taxpayer pointed 
out that some of the options were household names while others are not but that nothing 
indicated whether or not the options were the same quality. This taxpayer said, “I guess you 
could assume they are all equally valid.. .There is nothing on the results page that tells you that 
you can use any of them and get the same result, it’s legitimate and ok.” 

Suggested Solutions 

On the “Free File software offers” page (https://apps.irs.gov/app/freeFile/ and shown in Figure 
9), immediately above the boxes with the Free File software offers, include a statement like the 
following: “Below you will find the Free File software offers listed. The order of the offers is 
random and changes each time the page is closed and opened again.” 

Alternatively, the statement could say that the ordering does not indicate any quality difference 
between Free File software offers or any preference on behalf of the IRS and direct the user to 
choose the offering that best fits their individual needs. The arrow in Figure 9 shows where this 
statement could be placed. 
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Free File Software Offers 

If your adjusted gross income was $66,000 or less, you will find one or more free software options only available through Free File at IRS.gov. 
Review each company's offer to make sure you qualify for your free federal return. 

Some companies offer free state tax returns; some charge a fee. For more information, see software offers below and detailed information on 
company websites. The following states provide a free state tax return in partnership with private software companies via the Free File Alliance: 
AR, AZ, DC, GA, IA, ID, IN, KY, MA, Ml, MN, MO, MS, MT, NC, ND, NY, OR, Rl, SC, VA, VT, WV. Most of the tax software companies below provide a free 
federal return to active military personnel with an AGI of $66,000 or less. 

Browse the list of software offers below or use our tool. 


© Free File Software Lookup Tool 

Need help choosing Free File Software? Answer a few simple questions on income, age, state residence and more to find out which 
Free File software you can use. 


LOOKUP TOOL 


Free File Software Offers 



OnlineTaxesatOLT.com FreeTax Returns.com 


Figure 9. Free File Software Offers Page. 


4.2.5 Taxpayer Usability Issues while Performing Task 2b - Selecting a Free File 
Software Offer (With the Lookup Tool) 

In this task, the interviewers instructed taxpayers to use a fictitious scenario to select a Free File 
software offer using the Free File software lookup tool. 

4.2.5.1 Select Offer With Lookup Tool 1 


Table 22. Select Offer With Lookup Tool 1. 


Task 

Website 

Page/Section 

Issue 

Severity Count 

Select an 
Offer 
with the 

IRS.gov 

Lookup Tool 

Taxpayers assumed the 
lookup tool checks veracity 

High 5 

Lookup 

Tool 



of responses 



Description of the Issue 

Taxpayers assumed the lookup tool would perform a check of the veracity of their responses to 
the tool’s questions. They expected that the IRS tool would check their answers for accuracy. 

One taxpayer said, “I’m assuming that since I was routed through IRS.gov, the way I came into it 
specifically, that yeah I qualify [for the EITC], especially since I answered the questions 
correctly.... The IRS has a responsibility to make sure that certain items are correct.” 

Multiple taxpayers selected “yes” for being eligible for the EITC because they expected the 
website would verify their selections later. Taxpayers also chose “yes” to the EITC question 
because they did not want to miss an opportunity for a credit, and they believed the lookup tool, 
or another system would check their answers later. They did not think there were negative 
consequences to incorrect affirmative answers, only the potential negative consequence of 
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missing out on opportunities of the incorrectly answered in the negative. When asked by the 
interviewer why they selected “yes” for the EITC, one taxpayer responded, “I figured if it was a 
credit, they would check my info. If I said no, I thought I might miss out on an opportunity to see 
if I’m eligible.” Another echoed this sentiment, stating (regarding EITC), “There will be some 
kind of check and balance that will let me know I don’t make it... at some point it will check my 
work." 

This thinking resulted in taxpayers quickly making incorrect selections in the tool, particularly 
for the question about EITC eligibility. 

This issue is related to Select Offer with Lookup Tool 2. 

Suggested Solutions 

Include a statement in the tool description that says “Please enter your information accurately. 
The tool does not verify your responses.” Or “The tool does not confirm you qualify for certain 
credits but makes suggestions for software offers based on the information you enter. It does not 
guarantee that the Free File software offers suggested meet your specific needs.” 

Ensure that explanatory information, like the EITC pop-up modal, are easy to understand and 
support decision making. Some participants had difficulty understanding the EITC modal 
eligibility information and answered incorrectly after reading it. 

Alternatively, instead of asking about EITC eligibility, instead ask the taxpayer for information 
that determines EITC eligibility, like income (already asked), anticipated filing status, and 
number of children. This adds just one item to the tool’s questions. These questions are easier for 
taxpayers to correctly and quickly answer. 

On the tool results page, explicitly state that these recommendations are based on the user’s 
responses in the tool, and that the tool did not check their responses for accuracy. 

This issue is related to Select an Offer with Lookup Tool 11. 

4.2.5.2 Select Offer With Lookup Tool 2 


Table 23. Select Offer With Lookup Tool 2. 


Task 

Website 

Page/Section 

Issue 

Severity 

Count 

Select an 
Offer 
with the 

IRS.gov 

Lookup Tool 

Taxpayers selected "yes" to 
the EITC question when 

High 

8 

Lookup 

Tool 



unsure of their answer 




Description of the Issue 

When taxpayers were not sure about the correct response to EITC eligibility question in the 
lookup tool, they tended to select "yes." They erred on the side of wanting to maintain eligibility 
if possible and did not want to be discounted for opportunities. One taxpayer said, “I did it just in 
case.” 

Another taxpayer chose “yes” because they expected another check on their eligibility to occur 
later. They said, “I figure if I say yes, and I’m not, maybe they’ll tell me.” 
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Some taxpayers decided to choose “yes” based on their own EITC understanding or based on 
their own eligibility. 

Suggested Solutions 

Clarifying the purpose of the lookup tool and the fact that it does not check whether or not 
taxpayers’ responses are correct will help mitigate this problem. 

See suggested solutions for Select an Offer with the Lookup Tool 1 and Select an Offer with the 
Lookup Tool 11. 

Alternatively, consider removing the EITC question from the lookup tool entirely and including 
instead a series of questions that determine EITC eligibility for the taxpayer, as described in the 
suggestions for Select an Offer with the Lookup Tool 1. 

Make sure that the lookup tool and all associated descriptions and directions use plain language. 

4.2.S.3 Select Offer With Lookup Tool 3 


Table 24. Select Offer With Lookup Tool 3. 


Task 

Website 

Page/Section 

Issue 

Severity 

Count 

Select an 
Offer 
with the 
Lookup 
Tool 

IRS.gov 

Lookup Tool 

Taxpayers did not know how 
to answer the EITC question 
and wanted to leave it blank 
or leave the Lookup Tool 

High 

2 


When taxpayers arrived at the Earned Income Tax Credit question in the lookup tool (Figure 10), 
they expressed general confusion. One taxpayer did not know how to answer the question, ended 
up not finishing the lookup tool and did not select a Free File software offer. Another taxpayer 
wasn’t sure how to answer, decided to guess, and selected “yes.” 

Are you eligible for the Earned Income Tax Credit ?* 

O Yes 

O No 

Figure 10. EITC Question in Free File Software Lookup Tool. 


Possible Solutions 

Remove the requirement that taxpayers must select a yes or no response to the EITC eligibility 
question on the lookup tool. Accordingly, if a response is not provided, suggest Free File 
software offers that match both EITC eligible and ineligible in the suggested Free File software 
offer results. Visually separate these offerings, describe why they are separated, and enable the 
taxpayer to understand at this point whether they are eligible for the EITC or not. 

Consider replacing the single EITC eligibility question with multiple questions that determine 
EITC eligibility when taken together—filing status, AGI, wage income, and number of 
dependents. 

Make sure that the lookup tool and all associated descriptions, directions, and explanatory 
information use plain language. 
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These suggestions overlap with several in Select Offer with Lookup Tool 1 and Select Offer with 
Lookup Tool 2. 

4.2.54 Select Offer With Lookup Tool 4 


Table 25. Select Offer With Lookup Tool 4. 


Task 

Website 

Page/Section 

Issue 

Severity Count 

Select an 
Offer 



Taxpayers did not 
understand information in 


with the 
Lookup 
Tool 

IRS.gov 

Lookup Tool 

EITC modal pop-up, 
specifically misreading filing 
status requirements 

High 3 


Description of the Issue 

Taxpayers misread the text in the EITC modal and believed the EITC is for married people only. 
One taxpayer said, “He is not married so he wouldn’t be eligible.” 

Referring to the EITC modal, another taxpayer said, “It says for married but doesn’t say anything 
for single.” 

After reading the EITC modal, one taxpayer said, “Looks like he doesn’t qualify cause you gotta 
be married. That’s really unfair to single parents.” 

The modal does not explicitly mention any other filing statuses besides married filing jointly. 

The language in the modal leads taxpayers to believe that only married taxpayers are eligible for 
the EITC. 


Earned Income Tax Credit (EITC) * 

The Earned Income Tax Credit (EITC) is a refundable tax credit for low to moderate income working 
individuals and families. Taxpayers who earned less than $54,884 in 2018 may qualify for EITC. 

Earned Income and adjusted gross income (AGI) must each be less than: 

• $49,194 ($54,884 for married filing jointly) with three or more qualifying children 

• $45,802 ($51,492 for married filing jointly) with two qualifying children 

• $40,320 ($46,010 for married filing jointly) with one qualifying child 

• $15,270 ($20,950 for married filing jointly) with no qualifying children 

Use the IRS.gov EITC Assistant to see if you qualify. Taxpayers must file a federal income tax return, 
even if they are not required to file, and claim the credit to receive it. Some states have a similar 
credit. For additional information about the Earned Income Tax Credit (EITC), search IRS.gov for 
EITC Questions and Answers. 


Figure 11. EITC Pop-up Modal from the Lookup Tool. 


Suggested Solutions 

Split the EITC modal into two sections—one with information for married filing jointly and one 
for all other filing statuses. Additionally, list out the other filing statuses. 
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Alternatively, list the requirements with the dependent information up front and the two AGI cut¬ 
offs as distinct bullet points beneath each. This is suggested because users will read the 
beginning of the bullet points first and may not finish reading the points; and because it is 
generally easier and faster to accurately recall how many children one has than one’s earned 
income or AGI. Therefore, organizing around number of children enables the taxpayer to move 
faster through the dependency flow. For example: 

“Look below to find the cutoff for earned income and adjusted gross income (AGI) for each 
situation. If you are below the cutoff for your filing situation, you qualify for the EITC. 

• If you have no qualifying children 

o If you are married and filing jointly, less than [amount] 
o If you are NOT married and filing jointly: less than [amount]” 

• If you have one qualifying child 

o If you are married and filing jointly: less than [amount] 
o If you are NOT married and filing jointly: less than [amount] 

• If you have two qualifying children 

o If you are married and filing jointly: less than [amount] 
o If you are NOT married and filing jointly: less than [amount] 

• If you have three or more qualifying children 

o If you are married and filing jointly: less than [amount] 
o If you are NOT married and filing jointly: less than [amount] 

4.2.5.5 Select Offer With Lookup Tool 5 


Table 26. Select Offer With Lookup Tool 5. 


Task 

Website 

Page/Section 

Issue 

Severity 

Count 

Select an 
Offer 
with the 

IRS.gov 

Lookup Tool 

In general, taxpayers do not 
understand information in 

High 

3 

Lookup 

Tool 



EITC modal 




Description of the Issue 

Taxpayers read the text in the EITC modal yet inexplicably selected the incorrect answer for 
their scenario. One taxpayer believed they found the information they needed in the modal and 
said, “So basically, it tells you all the things you need to know if you qualify for earned income 
credit if you click on the bar for earned income credit.” This taxpayer still selected the incorrect 
answer. 

Suggested Solutions 

Do not assume that taxpayers are familiar with AGI, EITC, and how they are applicable to them. 
Use plain language to describe what they are and why they might matter to the taxpayer. Avoid 
extraneous information. 

Organize information clearly and use visual separation as appropriate. Do not “hide” important 
information in dense blocks of text. As much as possible, limit bullet points to one line long and 
do not nest clauses within a single bullet point. 
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Consider replacing the modal with a short wizard that walks the taxpayer through a few 
questions and then answers whether or not they are eligible for the EITC. If the EITC assistant is 
used here, modify it so that the user does not have to leave the page to walk through the assistant. 

See other suggestions in Select Offer with Lookup Tool 1, 2, 3, 4, and 6. 

4.2.5.6 Select Offer With Lookup Tool 6 

Table 27. Select Offer With Lookup Tool 6. 


Task 

Website 

Page/Section 

Issue 

Severity 

Count 

Select an 
Offer 
with the 
Lookup 
Tool 

IRS.gov 

Lookup Tool 

Taxpayers did not 
understand the relationship 
between having dependents 
and qualifying for EITC 

High 

2 


Description of the Issue 

Taxpayers had a preconceived notion that people with dependents automatically qualify for the 
EITC. Two taxpayers expressed this belief and committed an error and incorrectly selected yes 
on the lookup tool EITC question. One of the two committed this error despite reading the EITC 
modal in the lookup tool. 

Suggested Solutions 

Reverse the order of the bullet points on the EITC modal to start with no qualifying dependents. 
Bold or capitalize the word “no” to highlight the fact that taxpayers with no dependents can 
qualify as well. See the suggested solution for Select Offer with Lookup Tool 4 for an example of 
how this could look. 

Break out the statement “Taxpayers who earned less than $54,884...” into its own space and give 
it visual emphasis, e.g., by bolding it. Follow it with the detail, “Income limits depend on filing 
status and number of dependents. See below.” 

Suggestions in other EITC-related issues Select Offer with Lookup Tool 1, 2, 3, 4, and 5 apply as 
well, such as limiting the content per bullet point, structuring head bullet points to begin with 
number of dependents and replacing the EITC modal with a wizard. 

4.2.5.7 Select Offer With Lookup Tool 7 


Table 28. Select Offer With Lookup Tool 7. 


Task 

Website 

Page/Section 

Issue 

Severity 

Count 

Select an 
Offer 
with the 

IRS.gov 

Lookup Tool 

Taxpayer assumed incorrect 
criteria for EITC eligibility 
and did not seek additional 

High 

1 

Lookup 

Tool 



information to confirm 
assumption 




Description of the Issue 

The taxpayer assumed incorrect criteria for EITC eligibility and did not seek additional 
information from sources such as the lookup tool EITC modal to confirm their assumption. This 
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resulted in the taxpayer committing an error for EITC in the lookup tool. The taxpayer explained 
their decision to select yes for EITC eligibility: “It was just... your income, and if you have 
dependents or not, you would know that about yourself." 

Suggested Solutions 

Replace the single EITC question, which requires taxpayers to either understand EITC eligibility 
or read and understand the text in the EITC modal, with a series of questions that are designed to 
determine EITC eligibility (e.g., ask number of dependents, wage income, and filing status). This 
would remove taxpayers’ risk of believing they understand EITC eligibility when they do not, 
and thus responding inaccurately. 

Include a customer service or help function in the lookup tool page that does not require the user 
to leave the page, e.g., an FAQ section that can expand/hide or a chatbox to ask questions of a 
customer service representative or an automated customer service bot. 

Move some of the modal information into the lookup tool page itself and reword it more simply 
so users can make decisions more efficiently and accurately. 

4.2.5.8 Select Offer With Lookup Tool 8 


Table 29. Select Offer With Lookup Tool 8. 


Task 

Website 

Page/Section 

Issue 

Severity 

Count 

Select an 
Offer 
with the 

IRS.gov 

Lookup Tool 

Taxpayers were still 
confused about EITC after 

High 

2 

Lookup 

Tool 



using the EITC assistant 




Description of the Issue 

Taxpayers attempted to get more information about the EITC by using the EITC assistant 
(https://www.irs.gov/credits-deductions/individuals/eamed-income-tax-credit/use-the-eitc- 
assistant). Despite using this tool, they were still confused about EITC. Expressing their 
confusion, one taxpayer said, “The EITC tool was really frustrating. I got confused because it 
said something about married or widow that was confusing, and the result gave me widower 
[which did not apply to me]." 

Suggested Solution 

Providing solutions to improve the functionality of the EITC assistant is outside the scope of this 
assessment. However, MITRE recommends clarifying the language in the EITC modal, replacing 
the modal with an in-page wizard, or replacing the EITC eligibility question in order to lessen 
taxpayers’ need to exit the lookup tool and visit the EITC Assistant. 

See Select Offer with Lookup Tool 1, 2, 3, 4, 5, and 6 for related suggested solutions. 
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4.2.5.9 Select Offer With Lookup Tool 9 

Table 30. Select Offer With Lookup Tool 9. 


Task 

Website 

Page/Section 

Issue 

Severity 

Count 

Select an 
Offer 
with the 
Lookup 
Tool 

IRS.gov 

Lookup Tool 

Taxpayer did not understand 
income limits information in 
the EITC modal in the 
lookup tool 

High 

1 


Description of the Issue 

The taxpayer only reads the beginning portion of the information provided in the EITC modal 
(see Figure 11) in the lookup tool that states the income limit is $54,000. The taxpayer does not 
continue reading past this point, selects yes for EITC eligibility and says, “because it’s less than 
54K.” 

This selection was incorrect given the fictitious scenario the taxpayer was using. 

Suggested Solutions 

Replace the EITC eligibility question with questions that together assess EITC eligibility 
(income, qualifying dependents, filing status) but are individually easier to answer for taxpayers 
not well-versed in tax terminology. 

Re-arrange the contents of the EITC modal so that it is easier to read and understand at a glance. 
Visually separate different pieces of information and do not combine multiple pieces of 
important information into a single bullet point. Consider structuring the EITC modal content 
around number of qualifying dependents. 

Replace the EITC modal with a short, on-page wizard that walks the user through determining 
their EITC eligibility. 

See related suggestions in Select Offer with Lookup Tool 1, 2, 3, 4, 5, and 6. 

4.2.5.10 Select Offer With Lookup Tool 10 


Table 31. Select Offer With Lookup Tool 10. 


Task 

Website 

Page/Section 

Issue 

Severity 

Count 

Select an 
Offer 
with the 
Lookup 
Tool 

IRS.gov 

Lookup Tool 

Taxpayers were confused by 
missing states or selecting no 
state at the "state return" 
drop-down box 

Medium 

6 


Description of the Issue 

Taxpayers did not understand why all states were not shown in the state return drop-down box 
(Figure 12). Some taxpayers did not know how to select no state. Several taxpayers were 
confused by this but did not commit an error as a result. 
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One taxpayer selected the tooltip on state return question, but the state return popup box did not 
clarify the issue for the taxpayer (Figure 13). Another taxpayer thought the missing state might 
have been a computer blip, so they left the lookup tool page and looked for a Free File software 
offer from the list on the Free File software offers page. 


If you need to prepare and e-file a free state return , please select your 
state, otherwise click continue. 



Figure 12. State Return Question From the Lookup Tool. 


State Return * 

Select the state if you want to use the same company to prepare and e-file your state return for 
FREE as you plan to use for your federal return. Free File Inc. member companies have indicated 
they will provide free state tax return services to any state listed here, all of whom have agreed to 
Free File program rules. 

If you do not want to file your state return through this program or if you do not have a state income 
tax filing requirement, then skip this question. 

Seven states do not have a tax on wages: Alaska, Florida, Nevada, South Dakota, Texas, Washington, 
and Wyoming. Filings of State returns are not available for these seven states through the Free File 
program. 

Note: Fees for state tax returns may apply; however, some companies offer Free File state tax return 
preparation and e-filing. 


Figure 13. State Return Pop-up From the Lookup Tool. 


Suggested Solutions 

After the statement, “If you need to prepare and e-file a free state return, please select your state, 
otherwise click continue.” include an additional statement on a new line saying, “[Names of 
states] do not have a tax on wages, so they do not appear on the response options. [New line.] If 
your state does have an income tax and does not appear in the response options, your state is not 
eligible for free filing. You may still be eligible to file your federal returns through Free File in 
this case.” 

Use plain language in the state and residence information modals and explicitly address co mm on 
points of confusion. “Tax on wages” is inconsistent with “file your state tax return.” “Select the 
state, [etc.]... that you will include on your federal income tax return” from the residence modal 
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does not guide the taxpayer on what to do in cases of residing in multiple states, territories, etc. 
in the same tax year. 

Describe “residence” as “primary residence from Jan/year to Dec/year” in the residence question 
for increased clarity, accuracy, and efficiency. 

A series of alternative solutions are as follows: 

Include a “no state needed” option on the menu and revise the sentence to say, “If you need to 
prepare and e-file a free state return, please select your state, otherwise select ‘no state needed’ 
and click continue.” 

Another option is to include all states in the response drop down, and when a user selects a state 
that is not eligible for Free File, show an alert above, below, or next to the question area that 
says: 

• “This state is not eligible for Free File because it does not have a tax on wages. You do 
not need to file a state return for [state name]. You may still be eligible to file your 
federal return for free through Free File.” 

Or says: 

• “This state is not eligible for Free File. [State name] does have a tax on wages and you 
may need to file a state return for [state name] through another avenue. You may still be 
eligible to file your federal return for free through Free File.” 

This solution would greatly decrease user confusion and frustration. The question should still 
have a “Do not wish to prepare and e-file a state return” response and should not be required. 

Another option is to remove the state return question and use the taxpayer’s response on their 
state residence. This solution is suggested with the assumption that taxpayers typically only need 
to file state returns in the state of their residence, and with the observation that participants 
tended to look for a Free File software offer that supported state return filing when one was 
available. For this option, if the state residence is not Free File-eligible because it does not have a 
tax on wages, the lookup tool would return a statement along with the options that would say: 

• “[State name] is not eligible for Free File because it does not have a tax on wages. You 
do not need to file a state return for [state name]. You are still eligible to e-file your 
federal return for free through Free File using the Free File software offers named 
below.” 

And if the state does have an income tax but is not eligible for Free File, the statement would 
say: 

• “[State name] is not eligible for Free File. [State name] does have a tax on wages and you 
may need to file a state return for [state name] through another avenue. You are still 
eligible to e-file your federal return for free through Free File using the Free File software 
offers named below.” 

This solution would greatly decrease user confusion and frustration and cut down on the 
questions the taxpayer needs to answer using the tool. It still gives them the opportunity to 
choose a federal return-only offering. 
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4.2.5.11 Select Offer With Lookup Tool 11 


Table 32. Select Offer With Lookup Tool 11. 


Task 

Website 

Page/Section 

Issue 

Severity 

Count 

Select an 
Offer 
with the 

IRS.gov 

Lookup Tool 

Taxpayers were confused 
about the purpose of the 

Medium 

4 

Lookup 

Tool 



lookup tool 




Description of the Issue 

Taxpayers were not sure about the purpose of the lookup tool. The lookup tool button label and 
the supporting text do not clearly convey the purpose of the tool, which causes taxpayers to be 
confused about its functionality and to ascribe characteristics to the tool that it doesn't have. For 
example, taxpayers believed that the lookup tool would verify their eligibility for Free File, 
rather than help them select a Free File software offer they are eligible for based on their self- 
reports. One taxpayer explained that she believed the lookup tool was a “qualification” tool, “not 
a lookup tool; lookup tool makes me think I am already qualified... not trying to get qualified.” 

Suggested Solutions 

Change the button label text to task-oriented language such as, “Select Free File program for 
your tax return. 

Remove text that reads “Eligibility Verification” at the top of the lookup tool page 
(https://apps.irs.gov/app/freeFile/jsp/wizard.jsp?). This contradicts the concept of selecting a 
Free File software offer and reinforces the idea that the lookup tool verifies whether a taxpayer is 
eligible for Free File. (See Figure 14.) 

Include a new sentence immediately after the title on the lookup tool page 
(https://apps.irs.gov/app/freeFile/jsp/wizard.jsp). This title should say “The purpose of the Free 
File software lookup tool is to help you find a Free File software offer that you are eligible for. 
Try to answer questions correctly to the best of your ability; the tool does not check if your 
answers to eligibility questions are correct, and your answers impact the offers you are eligible 
for.” 

On the tool results page, the blue links appear in the same style as the rest of the IRS.gov site, 
making it seem like the user will be navigating within IRS.gov. Include logos to show the 
credibility of these non-IRS providers and to communicate that the user will be navigated away 
from the page. Have the offering open in a new browser window or new browser tab. 

On the tool results page, provide links back to other IRS.gov pages (e.g. tillable forms, IRS.gov 
home, browse all Free File offerings). Update the IRS Online Design Guide style so that users 
may access key pages ofIRS.gov. 
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SHIRS 


Free File Software Offers 


Free File Software Lookup Tool 


Complete the answers to determine what Free File software is available for you. If you cannot answer the questions, you may want to view the 
complete Free File Software Offers and services provided. 

All fields with * are required. 

Eligibility Verification 

Figure 14. "Eligibility Verification" Text on Lookup Tool Page Misrepresents Purpose. 


4.2.5.12 Select Offer With Lookup Tool 12 

Table 33. Select Offer With Lookup Tool 12 



Description of the Issue 

Taxpayers (many of whom expressed satisfaction with the lookup tool after using it later in the 
session) did not notice the tool on their own. They often overlooked the lookup tool box and 
quickly scrolled to the Free File software offers list. See Figure 15 for the lookup tool box; and 
Table 33 for task issue severity. 

Even taxpayers who had been to the Free File landing page before the session commented that 
they had never noticed the lookup tool on previous page visits. 

Prior to the interviewer directing them to the lookup tool, one taxpayer described a set of 
functions they had hoped for, which were similar to the functions of the tool. This taxpayer said, 
“I would like to have it [IRS site] ask what state I live in and it provide appropriate results. Can 
ask three questions to reduce the number of options: Is your income less than - and click the 
right income level. What state do you live in? And is your age less than 56?” 

The visual display of the lookup tool resembles an alert box, which means that when site visitors 
arrive on the page and skim its contents, they are less likely to interpret it as a tool they can use 
and more likely to read the title, decide they don’t understand what it does, and move down the 
page. 
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| An official website of the United States government 

1 

SHIRS 

Free File Home 


Free File Software Offers 


If your adjusted gross income was $66,000 or less, you will find one or more free software options only available through Free File at IRS.gov. 
Review each company’s offer to make sure you qualify for your free federal return. 

Some companies offer free state tax returns; some charge a fee. For more information, see software offers below and detailed information on 
company websites. The following states provide a free state tax return in partnership with private software companies via the Free File Alliance: 
AR, AZ, DC, GA, IA, ID, IN, KY, MA, Ml, MN, MO, MS, MT, NC, ND, NY, OR, Rl, SC, VA, VT, WV. Most of the tax software companies below provide a free 
federal return to active military personnel with an AGI of $66,000 or less. 

Browse the list of software offers below or use our tool. 


0 Free File Software Lookup Tool 

Need help choosing Free File Software? Answer a few simple questions on income, age, state residence and more to find out which 
Free File software you can use. 


LOOKUP TOOL 


Figure 15. Access to the Lookup Tool From Free File Software Offers Page. 


Suggested Solutions 

Move the lookup tool button to the top of the page, under the page header. Relabel the blue 
button to “Select a Free File Software Offer” to use task-oriented language and clarify its 
purpose. Use “website,” “web,” “online,” or “e-filing” rather than “software,” since users may 
not understand what is meant here by software and since many of the offers are for online filing. 
Change the header text “Free File Software Lookup Tool” to “Need helping choosing a Free File 
online offer?” to more clearly highlight its functionality. 

Change the visual display of the lookup tool box to resemble an alert less, perhaps by removing 
the “info” icon. Keep it separated from the general descriptive Free File information. 

Alternately, place a walk-through in the lookup tool box space on this page that walks the user 
through answering eligibility criteria and then highlights the eligible Free File software offers on 
the page. This minimizes mouse clicks and keeps all interactions centrally located to one area. 

4.2.5.13 Select Offer With Lookup Tool 13 


Table 34. Select Offer With Lookup Tool 13. 


Task 

Website 

Page/Section 

Issue 

Severity Count 

Select an 
Offer 
with the 

IRS.gov 

Lookup Tool 

Taxpayers perceived too 
many options returned from 

Low 3 

Lookup 

Tool 



the lookup tool 



Description of the Issue 

Taxpayers were overwhelmed by the number of choices returned from the lookup tool and had 
trouble choosing an offer. They did not understand why there were so many options and 
expressed that it would be difficult to make a choice. 
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Several taxpayers incorrectly selected yes for EITC eligibility, which resulted in additional 
results being returned from the tool. It is important to note that taxpayers who do actually qualify 
for EITC and correctly use the lookup tool will receive a large number of results from the tool as 
well. They may find the amount of offers available to them overwhelming and may find it 
difficult to select a software offer. 

This issue is related to Select Offer with Lookup Tool 14. 

Suggested Solutions 

Include a tooltip with each of the resulting offers that when clicked, allows taxpayers to see the 
eligibility criteria associated with each offer. This would give taxpayers the option to revisit the 
offers’ eligibility criteria without putting too much information on the results page and without 
requiring taxpayers to leave the page. 

Additionally, include the logo for each of the results to help taxpayers become more comfortable 
with the legitimacy of all the results. 

Add an option to the tool results page, e.g., a button, to return to the lookup tool. If possible, it 
should return the user to the lookup tool with their information still populated in the fields. 

Make the text at the top of the tool results page larger, simpler, and clearer to direct users to 
browse the offerings available on the providers’ websites. 

Explain clearly what the difference is between the federal only and federal and state Free File 
software offer results and explain why a taxpayer might choose one over the other. Recommend 
what kind of things the user should consider in choosing a Free File software offer that fits their 
needs best. 

4.2.5.14 Select Offer With Lookup Tool 14 


Table 35. Select Offer With Lookup Tool 14. 


Task 

Website 

Page/Section 

Issue 

Severity Count 

Select an 
Offer 
with the 

IRS.gov 

Lookup Tool 

Taxpayers expected to see 
eligibility criteria returned 

Low 3 

Lookup 

Tool 



with lookup results 



Description of the Issue 

Taxpayers did not know how to select a Free File software offer returned from lookup tool 
without being able to review the eligibility criteria in the same format as is shown on the Free 
File software offers page. Taxpayers expressed that either they would not know what to do on 
the results page or that they needed to do additional research on the offers. 

This issue is related to Select Offer with Lookup Tool 13. 

Suggested Solutions 

The suggestion from Select Offer with Lookup Tool 13 applies here: Include a tooltip with each 
of the resulting offers that when clicked, allows taxpayers to see the eligibility criteria associated 
with each offer. This would give taxpayers the option to revisit the offers’ eligibility criteria 
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without putting too much information on the results page and without requiring taxpayers to 
leave the page. 

Additionally, include the logo for each of the results to help taxpayers become more comfortable 
with the legitimacy of all the results. 

Add an option to the tool results page (e.g., a button) to return to the lookup tool. If possible, it 
should return the user to the lookup tool with their information still populated in the fields. 

Make the text at the top of the tool results page larger, simpler, and clearer to direct users to 
browse the offerings available on the providers’ websites. 

Explain clearly what the difference is between the federal only and federal and state Free File 
software offer results and explain why a taxpayer might choose one over the other. Recommend 
what kind of things the user should consider in choosing a Free File software offer that fits their 
needs best. 

4.2.5.15 Select Offer With Lookup Tool 15 


Table 36. Select Offer With Lookup Tool 15. 


Task 

Website 

Page/Section 

Issue 

Severity Count 

Select an 
Offer 
with the 

IRS.gov 

Lookup Tool 

Taxpayer was confused by 
the two questions regarding 

Low 1 

Lookup 

Tool 



states in the lookup tool 



Description of the Issue 

One taxpayer did not understand why the lookup tool had two questions (Figure 16 and Figure 
17) about states and did not understand the difference between them. 


What is your residence for filing your 2018 income tax return?* 
Select Residence v 

Figure 16. Lookup Tool State Question 1 - Residence. 


If you need to prepare and e-file a free state return , please select your 
state, otherwise click continue. 

Select State v 

Figure 17. Lookup Tool State Question 2 - Filing a State Return. 


This is further complicated by the fact that not all states show up in the second state question, 
which is specifically about preparing and e-filing a free state return. For more information on this 
topic, see Select Offer with Lookup Tool 10. 
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Suggested Solutions 

Suggestions from Select Offer with Lookup Tool 10 apply here: 

After this statement—“If you need to prepare and e-file a free state return, please select your 
state, otherwise click continue.”—include an additional statement on a new line saying, “[Names 
of states] do not have a tax on wages, so they do not appear on the response options. [New line.] 
If your state does have an income tax and does not appear in the response options, your state is 
not eligible for free filing. You may still be eligible to file your federal returns through Free File 
in this case.” 

Use plain language in the state and residence information modals and explicitly address common 
points of confusion. “Tax on wages” is inconsistent with “file your state tax return.” “Select the 
state, [etc.]... that you will include on your federal income tax return” from the residence modal 
does not guide the taxpayer on what to do in cases of residing in multiple states, territories, etc. 
in the same tax year. 

Describe “residence” as “primary residence from Jan/year to Dec/year” in the residence question 
for increased clarity, accuracy, and efficiency. 

A series of alternative solutions are as follows: 

Include a no state needed option on the menu and revise the sentence to say, “If you need to 
prepare and e-file a free state return, please select your state, otherwise select ‘no state needed’ 
and click continue. 

Another option is to include all states in the response drop down, and when a user selects a state 
that is not eligible for Free File, show an alert above, below, or next to the question area that 
says: 

• “This state is not eligible for Free File because it does not have a tax on wages. You do 
not need to file a state return for [state name]. You may still be eligible to file your 
federal return for free through Free File.” 

Or says: 

• “This state is not eligible for Free File. [State name] does have a tax on wages and you 
may need to file a state return for [state name] through another avenue. You may still be 
eligible to file your federal return for free through Free File.” 

This solution would greatly decrease user confusion and frustration. The question should still 
have a “Do not wish to prepare and e-file a state return” response and should not be required. 

Another option is to remove the state return question and use the taxpayer’s response on their 
state residence. This solution is suggested with the assumption that taxpayers typically only need 
to file state returns in the state of their residence, and with the observation that participants 
tended to look for a Free File software offer that supported state return filing when one was 
available. For this option, if the state residence is not Free File-eligible because it does not have a 
tax on wages, the lookup tool would return a statement along with the options that would say: 

• “[State name] is not eligible for Free File because it does not have a tax on wages. You 
do not need to file a state return for [state name]. You are still eligible to e-file your 
federal return for free through Free File using the Free File software offers named 
below.” 
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And if the state does have an income tax but is not eligible for Free File, the statement would 
say: 

• “[State name] is not eligible for Free File. [State name] does have a tax on wages and you 
may need to file a state return for [state name] through another avenue. You are still 
eligible to e-file your federal return for free through Free File using the Free File software 
offers named below.” 

This solution would greatly decrease user confusion and frustration and cut down on the 
questions the taxpayer needs to answer using the tool. It still gives them the opportunity to 
choose a federal return-only Free File software offer. 

4.2.6 Taxpayer Usability Issues with Task 3 - Filing a Return Using a Pre-assigned 
Free File Software 

In this task, the interviewers instructed taxpayers to use a given set of fictitious information to 
complete a federal income tax return using a pre-assigned Free File Software. 

4.2.6.1 File a Return 1 


Table 37. File a Return 1. 


Task 

Website 

Page/Section 

Issue 

Severity 

Count 

Filing a 
Return 

Member 

3, 

Member 

11 

Student Loan 
Interest 

Taxpayers were uncertain 
about where to enter student 
loan interest information 

High 

7 


Description of Issue 

Taxpayers were not sure where to enter their student loan information from the Form 1098-E 
while filing a return using Member 3 and Member 11. Specifically, some taxpayers were looking 
for a place to enter it while they were entering information from the Form W-2 or while entering 
income information. The place to enter student loan interest information was not where they 
expected it to be. One taxpayer said, “I was looking for student loan info in the wrong place in 
income but it’s on the deductions and credits page under education.” 

On the page for entering Form W-2 information, for example, another taxpayer said, “Where’s 
the stuff for the student loan interest? Did I miss it? There’s no place to put the student loan 
interest unless I’m missing it.” 

When given the option to select the forms they had for filing, some taxpayers did not select 
1098-E from the list of documents, which prevented them from receiving any prompt from the 
system to enter student loan interest. 

This uncertainty resulted in some taxpayers neglecting to enter any student loan information. 
Therefore, they incorrectly completed the return. 

Suggested Solutions 

Include an education expenses section with a checkbox for student loan payments on the page 
where taxpayers are selecting the items of basic information such as income and dependents that 
apply to them. Checking the box for student loan payment should trigger the presentation of a 
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Student loan interest payment deduction page when taxpayers later enter their information for 
deductions. 

Make sure navigation is usable and descriptive to ensure users can go back and find where to 
enter student loan information if they miss it on their first pass. 

Alternatively, provide overview pages for income, deductions, and credits that list all the 
co mm on items and forms and allow taxpayers to select and enter information for them. 

4.2.6.2 File a Return 2 


Table 38. File a Return 2. 


Task 

Website 

Page/Section 

Issue 

Severity Count 

Filing a 

Member 

3, 

Select a 

Taxpayers were uncertain 
about selecting a filing status 

High 5 

Return 

Member 

Filing Status 

of either head of household 


11 


or single 



Description of the Issue 

Taxpayers were uncertain or confused about selecting either head of household or single as their 
filing status. Some taxpayers assumed that since their marital status was single and they had one 
dependent in the scenario that they should file as head of household, which was not correct. 

Taxpayers expected more information within the system to help them make the correct choice. 
Instead, they had to turn to an external source for more information. One taxpayer said, “The 
head of household alert provided a list of things without much guidance to discern if it applies. I 
had to do a separate Google search to learn more.” 

If they choose head of household, some taxpayers expected to see questions or prompts to 
confirm their selection. One taxpayer who chose head of household said, “I thought it would ask 
some clarifying questions to make sure I was actually supposed to be filing as head of household, 
I don’t know why it didn’t ask me follow up questions. Maybe it did verify, and I didn’t realize 
it?” The Member 3 system does not provide any clarifying questions to confirm head of 
household qualifications. 

The Member 11 system did not provide a step to verify that the taxpayer paid over half of their 
household expenses and is therefore qualified as head of household. This characteristic is what 
defined the taxpayer as a single rather than head of household filer in the test scenario. 

This uncertainty led some taxpayers to make the incorrect selection of the head of household 
filing status. 

This issue is related to File a Return 12. 

Suggested Solutions 

Provide step-by-step prompts to help taxpayers make the correct filing status selection, 
particularly between single and head of household. 


54 

PRE-DECISIONAL - FOR INTERNAL IRS W&l USE ONLY - NOT FOR PUBLIC RELEASE 




PRE-DECISIONAL - FOR INTERNAL IRS W&l USE ONLY - NOT FOR PUBLIC RELEASE 


4.2.6.3 File a Return 3 

Table 39. File a Return 3. 


Task 

Website 

Page/Section 

Issue 

Severity 

Count 

Filing a 
Return 

Member 

11 

Quick File 

Taxpayers were confused 
about what to do once they 
chose the quick file option 

High 

3 


Description of the Issue 

After completing basic information, taxpayers could choose between the options “Quick File” or 
“continue” for filing a federal return. With quick file, taxpayers would have to enter in the names 
of their forms on the next page. With continue, taxpayers would be taken to what Member 11 
called a “more traditional style of entry.” 

Some of the taxpayers who chose quick file became confused about what to do next once they 
chose quick file. On the quick file page, taxpayers are supposed to type in the forms they have 
and then proceed to the next pages to enter the information from the forms. They were 
immediately stuck and did not know what to do. 

One taxpayer expected the quick file option to be simple since they believed the scenario 
information was simple. They said, “I would click on the quick file. It’s pretty clear cut. She’s 
not self-employed or anything. I would just go to quick file to see what I needed to do.” 

This same taxpayer did not like that they were supposed to enter the forms they needed. This 
taxpayer said, “I actually don’t like this, it’s asking me what forms I need and I’m not sure, I’m 
going to go back and go with the other option.” This taxpayer backed out of quick file and was 
taken to where they selected “guide me.” 

Another taxpayer expected there to already be information provided in the system. They said, “I 
thought since all the info was there, she was a citizen, I don’t have to go through every tiny 
detail. I was sure she did taxes last year so I figured her info was there.” 

This taxpayer attempted to work through quick file but spent a lot of time going through pages 
that were not relevant to their scenario. This taxpayer eventually made it to the completed return 
page without entering any income information from their Form W-2. 

Suggested Solution 

Change the name of quick file to something that lets taxpayers know the option requires a fairly 
high level of familiarity with their forms and tax situation. The name “quick file” leads taxpayers 
to believe that it is an easier option than choosing to continue for the “more traditional style of 
entry.” It also may lead users to think that it involves some sort of file upload or employer 
identification number (EIN) document look-up, which it does not. 
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4.2.6.4 File a Return 4 

Table 40. File a Return 4. 


Task 

Website 

Page/Section 

Issue 

Severity 

Count 

Filing a 
Return 

Member 10, 
Member 11 

Create an 
Account 

Taxpayers faced difficulty 
creating an account 
password 

High 

4 


Description of the Issue 

Taxpayers had a difficult time creating a new account password that met all password 
requirements (e.g., length, use of special characters, etc.). 

Taxpayers did not find the password requirement information clear enough or they did not notice 
it. One taxpayer said, “They should have been more explicit in setting up the password 
(uppercase, lowercase, using symbols), so I knock off a point for that.” 

Another taxpayer who did not notice the password requirements until they created an ineligible 
password said, “I don’t like that it didn’t tell you what’s required in the password until after you 
have entered it. I don’t like the way the password stuff is set up here. I want see the requirements 
before I type it in.” 

All taxpayers but one participant eventually was able to overcome this challenge and proceed 
through the task; however, one taxpayer became visibly and vocally frustrated while trying to set 
up a password and was unable to continue on their own. A researcher had to intervene to assist 
the taxpayer through the password setup. 

Suggested Solutions 

Show password requirements in bold and brightly colored text before, during, and after taxpayers 
are entering a new password. Use an icon as well as different text colors to indicate whether the 
entered password meets requirements or not. Update this feedback in real time as the password is 
entered into the field. 

Let users see what they have typed into the password field (a show/hide function may work well 
here). 

It is recommended to show a real-time strength meter to encourage taxpayers to create strong 
passwords. 

Follow user experience best practices in designing password interactions. 

4.2.6.5 File a Return 5 


Table 41. File a Return 5. 


Task 

Website 

Page/Section 

Issue 

Severity 

Count 

Filing a 
Return 

Member 

11 

Health 

Insurance 

Taxpayers were confused by 
the wording of the health 
insurance question 

Medium 

3 


Description of the Issue 
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Taxpayers incorrectly selected no when they arrived at the question asking if they had health 
insurance or were eligible for an exemption. This happened because participants were confused 
by the wording of the question. The question stated, “Did you or your family have health 
insurance or qualify for a health insurance coverage exemption at any time in 2018?” 

Taxpayers were confused by the either/or nature of the question. One taxpayer said, “With health 
insurance I was tripped up by that question, if they had broken it into two questions, I would’ve 
understood better.” 

One taxpayer was specifically confused by the word “exemption” in the question. This taxpayer 
said, “When asked about health insurance, I answered ‘no’ because I saw the word exemption in 
there. It should just say, ‘Did you or your family have employer sponsored health insurance?’ I 
find the question they have here a little confusing. It seems like they made it harder than it 
should be.” 

Despite their initial confusion, once they received an alert or error, taxpayers went back to 
correct their answer to yes. However, clarifying the question may lead taxpayers to answer 
correctly the first time. 

Suggested Solutions 

Use plain language and do not assume that users have in-depth knowledge about tax filing 
matters. For example, provide an explanation of what an exemption is and how one knows if 
they qualify, or rephrase the question to be understandable to users with a lower level knowledge 
of taxes. Clarify that having health insurance from the marketplace is different than having it 
through an employer; “the marketplace” is not every-day terminology and appears confusing. 

Separate the health insurance item into two separate questions that appear one at a time. On the 
first screen present taxpayers with the question, “Did you or your family have health insurance in 
2018?” If the response is yes, proceed to the next item. If the response is no, present taxpayers 
with the next question, “Did you or your family qualify for a health insurance coverage 
exemption in 2018?” 

4.2.6.6 File a Return 6 


Table 42. File a Return 6. 


Task 

Website 

Page/Section 

Issue 

Severity 

Count 

Filing a 
Return 

Member 

3 

Dependent 

Information 

Taxpayers were confused by 
the question asking how 
many months the dependent 
lived with them in the U.S. 

Medium 

2 


Description of the Issue 

Taxpayers were confused by the question asking how many months their fictional scenario 
dependent had lived with them. One taxpayer incorrectly entered 0 and proceeded. The 
participant did not explain why they made this selection. Another taxpayer skipped the field 
entirely. They were particularly confused by the “in the U.S.” portion of the question and 
confused by the icon and modal in the field for the number of months. This taxpayer explained 
their confusion saying, “I’ve never had to file a dependent, so this is all new to me”. 
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Suggested Solutions 

Split this question up to walk the taxpayer through simple questions one at a time. For example: 

• “Did your dependent live with you all 12 months of the year? 

o Yes: “Did you and your dependent live together outside of the U.S. at any point 
during the filing year? 

■ Yes: “How many months did you and your dependent live together outside 
of the U.S. during the filing year?” 

■ No: No further questions needed. 

o No: “How many months did your dependent live with you?” 

■ “Did you and your dependent live together outside of the U.S. at any point 
during the filing year? 

• Yes: “How many months did you and your dependent live together 
outside of the U.S. during the filing year?” 

• No: No further questions needed. 

To accommodate newer or less-experienced filers, include options (including being directed to 
relevant IRS informational pages) on what the benefits are of claiming a dependent, and what the 
parameters are around it. A modal could be used for this, or a help function where the taxpayer 
can type in questions. 

4.2.67 File a Return 7 


Table 43. File a Return 7. 


Task 

Website 

Page/Section 

Issue 

Severity 

Count 

Filing a 
Return 

Member 

3 

Create an 
Account 

Taxpayers faced difficulty 
creating an account 
username 

Medium 

3 


Description of the Issue 

Taxpayers faced difficulty creating an account username because they did not notice the 
requirements for a username. Taxpayers also became frustrated when they created a username 
that already existed. They then received errors saying either the username did not meet the 
requirements, or the name was already taken when they attempted to create a new username. One 
taxpayer tried several times to create a username that was not already taken and commented that 
they were not sure what the username requirements were. 

All taxpayers were eventually able to overcome this challenge and proceed through the task; 
however, it did cause frustration. One taxpayer said, “I was hung up on password and username. 
That was horrible." 

Suggested Solutions 

Show username requirements in bold and brightly colored text before, during, and after 
taxpayers are entering a new username. Use an icon as well as different text colors to indicate 
whether the entered username meets requirements or not, and whether it is available or already 
taken. Update this feedback in real time as the username is entered into the field. 
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Follow user experience best practices in designing usemame/password interactions. 
These solutions are related to those suggested in File a Return 4. 


4.2.6.S File a Return 8 

Table 44. File a Return 8. 


Task 

Website 

Page/Section 

Issue 

Severity 

Count 

Filing a 
Return 

Member 

11 

Use of 
Information 
Agreement 

Taxpayers attempted to enter 
the scenario taxpayer’s 
birthday instead of the 
current date 

Medium 

4 


Description of the Issue 

On the use of information agreement page, taxpayers attempted to enter the fictitious scenario 
taxpayer’s (Christine Coleman’s) birthday after entering her name when the system was actually 
asking for the current date. Taxpayers became confused when the system would not let them 
select a year other than 2019. The field descriptor only said “date,” which caused taxpayers to be 
confused about what date they were supposed to enter. Taxpayers realized their error, corrected 
them, and continued with the task. 

Suggested Solution 

Change “Date” to “Current Date” or “Today’s Date” to specify the date taxpayers are supposed 
to enter. 


4.2.6.9 File a Return 9 

Table 45. File a Return 9. 


Task 

Website 

Page/Section 

Issue 

Severity 

Count 

Filing a 
Return 

Member 

11 

Enter 

information 
from Form 
W-2 

Taxpayers incorrectly 
rounded Medicare tax 
withheld to the nearest dollar 

Low 

3 


Description of the Issue 

When entering information from the Form W-2, taxpayers incorrectly rounded the Medicare tax 
withheld to the nearest dollar (from $4.62 to $5.00). Once they finished entering information 
from the Form W-2, the system gave them an alert saying their Medicare withholding seemed to 
be too high. Taxpayers returned to the previous page and corrected the Medicare tax to the exact 
amount. 

Suggested Solutions 
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In the instructions for Form W-2 information, include a statement telling taxpayers to enter the 
exact amounts found on the Form W-2. Put a short specific instruction at every dollar-amount 
entry field telling the user to enter the exact amount. 

Tell the user in real-time if they have entered invalid information into any field through visually 
highlighting the field, using an alert icon and different colored text, and providing a small 
statement on the nature of the issue, e.g., “must be whole numbers.” 

4.2.6.10 File a Return 10 


Table 46. File a Return 10. 


Task 

Website 

Page/Section 

Issue 

Severity 

Count 

Filing a 
Return 

Member 

3 

Form W-2 
Quick Entry 

Taxpayers had difficulty 
selecting items in the Quick 
Entry W-2 

Low 

3 


Description of the Issue 

Taxpayers face difficulty and become frustrated selecting items (e.g., employer zip code and 
employer state) in the Form W-2 quick entry. Items such as employer zip code and employer 
state were not responsive to taxpayers' clicks. 

One taxpayer said, “I find the click ability of this page not super ideal, getting the mouse on 
things I don’t want it to go on, I’m not able to tab through.” 

Suggested Solutions 

Make the targets (items to click on) larger so that they are easier to select and click on (according 
to Fitts’s Law [3]). Allow the user to move between fields using the tab key on the keyboard. 
Following usability best practices, design the form so that users are able to focus on and edit 
every field with a keyboard only to allow for both power users and keyboard-dependent users 
[4], 

4.2.6.11 File a Return 11 


Table 47. File a Return 11. 


Task 

Website 

Page/Section 

Issue 

Severity Count 

Filing a 
Return 

Member 

10 , 

Member 

Overall 

Taxpayers were confused by 
the path or flow of 
information presented while 

Low 3 


11 


filing 



Description of the Issue 

Taxpayers did not always understand the path or flow of information presented to them while 
filing. They lost track of where they were in the filing process or did not see the pages they were 
expecting to see. 

One taxpayer said, “I feel like I click one thing and it gives me something and then I go back and 
click the same thing and it give me something different. I don’t understand the path sometimes.” 
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One taxpayer became confused about where they were while on the payments and estimates 
page. This taxpayer said, “I don’t know where I am right now, something with estimated 
payments. I don’t need to do any of these...” The taxpayer went back to payments and estimates 
page and said, “I’m just confused about where I am.” 

Another taxpayer attempted to return to the sales tax page to reenter receipt information but was 
taken to the standard deduction page instead. When the interviewer asked if they were expecting 
to see the standard deduction page, the taxpayer said, “No, I thought I would go back to sales tax 
but I guess it must have autocorrected itself since it seems to have gone to a standard deduction. 
So, I am assuming it corrected itself. And if it doesn’t want to know, I don’t have to tell it.” 

Suggested Solutions 

In the side navigation menu, use symbols and color saturation to indicate where in the process 
the taxpayer is and the status of the section. For example, use check marks to indicate that the 
taxpayer has completed a section and an empty circle to show incomplete sections, and use color 
or color saturation differences to highlight where in the menu the taxpayer currently is. 

Include a help function to aid users in finding specific sections. For example, a help function 
could be an area where the taxpayer types in what topic they are searching for (Form W-2, 
student loan interest, credits) and in turn receives a link or instructions to get to that part of the 
process. 

Conform to usability heuristics to ensure that system status is visible, that users know where they 
are in the workflow at all times, and that users are able to revisit and move forward seamlessly 
with the system autosaving their process as they navigate forms. 

4.2.6.12 File a Return 12 


Table 48. File a Return 12. 


Task 

Website 

Page/Section 

Issue 

Severity Count 

Filing a 
Return 

Member 

11 

Overall 

Taxpayer questioned their 
filing status selection after 
choosing to file as single 

Low 1 


Description of the Issue 

One taxpayer used the Member 11 system’s filing wizard to determine that their filing status 
should be single. The taxpayer later entered dependent information. After, the system gave them 
a prompt asking if they wanted to change their filing status to head of household. This prompt 
made the taxpayer lose confidence in the accuracy of the filing wizard and their selection of 
single. The taxpayer said, “Seeing the message made me question if I used the wizard correctly, 
otherwise the system wouldn’t have asked the question." 

The taxpayer clicked the “Click here to see if you qualify” link provided and verified filing status 
should be single. 

This issue is related to File a Return 2. 

Suggested Solution 
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In the prompt alert, explain in more detail why the suggestion was made (i.e., explain what 
qualifies someone for head of household), and affirm that the participant does not have to change 
their selection. Keep the qualification walk-through option. 

Yellow and red are culturally associated with mistakes and danger in online interfaces in 
Western culture. Change the color of the alert so it is not yellow or red, since these colors lead 
the user to immediately think they have made a mistake that must be corrected. 

Alternately, when a taxpayer uses the filing wizard to determine their status, do not give them 
any prompts to make changes to their filing status unless they later make selections or enter 
information that contradicts what they have entered into the filing wizard. For example, if a 
taxpayer indicates that their marital status is single, they have a dependent, and they do not pay 
for more than half of household expenses, do not later give them a prompt asking if they want to 
file as head of household. The filing wizard already determined they do not qualify due to paying 
for less than half of their household expenses. 

4.2.6.13 File a Return 13 


Table 49. File a Return 13. 


Task 

Website 

Page/Section 

Issue 

Severity Count 

Filing a 
Return 

Member 

3 

Basic Info - Life 
Events & 
Child/Dependent 
Care 

Taxpayer confused the 
check box for having 
dependent care expenses 
with the check box for 
having a dependent 

Low 1 


Description of the Issue 

On the Basic Info-Life Events page, a taxpayer misread the children/dependents and 
child/dependent care expenses checkboxes and erroneously checked that they had 
child/dependent care expenses. This selection created challenges for the taxpayer later on the 
child/dependent care expenses page. 

When the taxpayer arrived at the child/dependent care question later, the taxpayer read “Did you 
have dependent care expenses” and said, “I had dependents, and clicked yes.” The taxpayer 
became confused on the next child/dependent care page, realized they made a mistake, and 
attempted to back out of the section. However, the taxpayer had difficulty backing out and 
entered zero for dependent care expenses paid to get out of the section. Taxpayer returned to 
choose the correct answer “no.” 

Suggested Solutions 

Include the words “paid for” before “child/dependent care expenses” so that the taxpayer does 
not mistakenly conflate the two items as they skim the list. For consistency, include the words 
“paid for” before every expense item on the list (child/dependent care expenses, adoption 
expenses); it is already included for alimony expenses. 

Ensure that system navigation is improved according to the suggested solutions for File a Return 
11 in order to enable the user to efficiently correct mistakes and then return to their latest stage in 
the filing process. 
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5 User Experience Questionnaire Findings 

This section presents MITRE’s findings from user experience questionnaire data collected 
throughout the taxpayer testing sessions. The two questionnaires captured perceived 
usability and ease of use for discrete segments of the session (per task) as well as over the 
entire session experience. MITRE analyzed the questionnaire responses overall, as well as 
disaggregated by Free File member software offer used and by age and education level. 
Overall findings include: 

• Overall, usability scores indicate that the Free File system is acceptably usable, with 
an average system rating of a “B.” Participants more often agreed with the statement 
that the Free File system meets their requirements than the system is easy to use. 

• On average, participants found Tasks 1 and 2 to be easier than Task 3. 

• Ease of use and overall usability ratings varied by Free File member software used. 

• Patterns by education level were unclear; less-educated participants rated ease of use 
for Tasks 1 and 2 lower than more-educated participants, but rated overall usability 
of the system higher than more educated participants. 

• Participants in the lowest age tercile (20-35 years) reported greater ease of use and 
satisfaction than older participants. 


Participants rated their satisfaction and overall experience with the Free File process by 
responding to usability questionnaires. After completing each task, participants responded to a 
SEQ regarding that task. After completing all tasks, users answered the UMUX about their entire 
Free File experience during the session. 

The SEQ is a single-item questionnaire that measures user perception of ease related to specific 
tasks. It is asked directly after the relevant task and is administered on a seven-point response 
scale, where one is “very difficult” and seven is “very easy.” SEQ scores are interpreted against 
an industry-accepted scale of below-average, average, and above-average SEQ response ranges 
(see Sections 9A.3.3 and A.4.5). 

Item: “Overall, how easy or difficult was the task to complete?” 

The UMUX is a four-item questionnaire that looks at an entire experience; it is not task-specific. 
Its four questions for measuring usability are targeted at the International Organization for 
Standardization (ISO) 9241 definition of usability 12 . The following items are included in the 
UMUX questionnaire: 

• “[This system’s] capabilities meet my requirements.” 

• “Using [this system] is a frustrating experience.” 


12 ISO 9241-11:2018 defines usability as “the extent to which a system, product or service can be used by specified 
users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use.” 
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• “[This system] is easy to use.” 

• “I have to spend too much time correcting things with [this system].” 

There are two positive and two negative items, all administered on a seven-point response scale, 
where one is “strongly disagree” and seven is “strongly agree.” Free File software system was 
defined as “this system” in the UMUX questionnaire. The UMUX responses are converted to 
scores and compared to an industry-accepted acceptability and grading comparison system. See 
Sections A.3.3 and A.4.5 for more detail. 


5.1 Summary Statistics 

In this section, SEQ and UMUX scores are discussed together for each topic of comparison. In 
charts, SEQ and UMUX responses and scores are overlaid on response and score bands created 
by previous industry and academia work: below-average range, average range, and above- 
average SEQ range bands; and not-acceptable range, marginally acceptable range, and 
acceptable UMUX range bands. Section A.4.5 describes these bands in further detail. Due to the 
small sample size, MITRE did not conduct tests of statistical significance with these results; 
therefore, these results should not be interpreted as representative of the full free filer population. 


5.1.1 Overall Usability 

The SEQ responses collected were evaluated against the typical SEQ response bands. These 
bands are based on SEQ data from industry usability studies on a variety of interfaces, in which 
tests over 400 tasks and 10,000 users produced average scores of about 5.3 to 5.6 [5], In this 
study, 5.3 to 5.6 is treated as the average response band, with ratings under 5.3 and over 5.6 
treated as below- and above-average ranges, respectively (see Figure 18). Section A.4.5.1 
provides more detail on the SEQ scoring. 
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Figure 18. Free File Usability SEQ Scoring Bands. 


Figure 20 shows the average SEQ scores for all participants are 5.97, 6.0, and 5.10 for Task 1, 
Task 2, and Task 3, respectively. Task 1 required participants to locate the IRS Free File 
program using an internet browser, Task 2 required participants to use information provided 
within a factious scenario to identify and select an eligible Free File program offering, and Task 3 
required participants to complete a fictitious tax return using one of three Free File program 
offers randomly assigned by MITRE. 

Considering usability throughout the entire Free File experience, MITRE found that as a whole, 
participants perceived Tasks 1 and 2 as acceptably “easy” (~6) and rated Task 3 lower, giving it 
an average rating of 5.1. Counter to expectations based on these ratings, MITRE observed more 
usability issues with Tasks 1 and 2 (detailed in Section 4.2) than with Task 3. 
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Task 3 was more complicated than the other tasks and going through the e-filing process is 
inherently different from web navigation and web content comprehension. For this reason, 
ratings for Task 3 cannot be compared meaningfully to ratings for Tasks 1 and 2. Additionally, 
participants voiced general negative feelings about filing taxes and commented that it was more 
difficult filing using information that was not their own. Section 1.3 discusses the shortcomings 
of using artificial tax scenario data. 

Average SEQ Ratings Across Tasks 


5.97 6.00 



above average SEQ, >5.6 ~ average SEQ, 5.3-5.6 ^assays below average SEQ, <5.3 ^^Mcan SEQ Responsesj 

Figure 19. Average SEQ Ratings Across Tasks. 

The UMUX responses collected were converted into scores with values 0-100 using Finstad’s 
process [6]. The processed UMUX scores are then treated like System Usability Scale (SUS) 
scores and rated against a scale using the descriptors in Figure 20. See Section A.4.5.2 for more 
detail on the UMUX scoring process. 


SUS Score: 
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Figure 20. SUS Scale for Free File Program Usability Study. 
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Figure 21 shows the average UMUX scores for all participants is 77.2, meaning users rated the 
entire experience of all tasks combined at about a “B” grade. This is well within the acceptable 
usability range. Figure 22 shows the mean rating response for each question. On average, 
participants agreed more strongly that the system “meets my requirements” than that it was “easy 
to use,” suggesting that users may not expect ease when they file returns. There is also room for 
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improvement in making the experience less “frustrating” and cutting down on “too much time 
correcting mistakes.” 


Average UMUX Score 



Acceptable Marginal saaaaass Not Acceptable ^^—Mean UMUX Score 


Figure 21. Average UMUX Score. 
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Figure 22. Average UMUX Responses. 
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Eight of the 29 participants were uncertain if they had used the Free File program in one of the 
prior two years, or a free commercial service (See Section A.2.4). This implies unfamiliarity with 
the Free File Program prior to arriving to sessions. As they learned more about Free File’s free 
options through completing tasks, they seemed surprised and pleased about many aspects of the 
opportunities available to them (though they also expressed disappointment that they had not 
known about the opportunities earlier). Several remarked that they would use what they learned 
in the session when they filed next. These positive impressions may have played a part in 
positive responses about the entire Free File experience. 

5.1.2 Usability by Vendor Offering 

Figure 23 provides the average participant SEQ rating for Task 3, which required participants to 
complete a tax return using a Free File Offering assigned by MITRE. Member 10 received a 
relatively high average SEQ rating of 5.8. Member 3 (5.2) and Member 11 (4.3) were rated 
below average for ease of use. 


Task 3 Average SEQ Ratings by Vendor 
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Figure 23. Task 3 Average SEQ Ratings by Vendor. 

The UMUX responses provided in Figure 24 show a similar pattern. Member 10 and Member 3 
received “acceptable” scores of 90.3 and 74.2, respectively. Member 11 received a “marginal” 
score of 67.6. The shared pattern suggests that the usability of the Free File software offers made 
a strong enough impression to impact the overall Free File taxpayer experience. 
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Average UMUX Scores by Vendor 
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Figure 24. Average UMUX Scores by Vendor. 

Commonly observed usability issues were not knowing where to enter student loan information, 
difficulty picking filing status, and confusion about flow of use (e.g., navigating to review 
information or correct mistakes). These issues occurred with Member 3 and Member 11, not with 
Member 10, and may have caused the lower overall scores associated with these systems. Other 
common issues of where to enter health insurance information, understanding the “quick file” 
option, and addressing unclear error messages during Form W-2 entry were only with Member 
11. 

Fourteen participants reported having used Member 10 before while three participants said they 
had previously used Member 3, and only one reported having used Member 11 in the past 
(Section 6.3). Familiarity can contribute to the experience of usability, so software provider 
familiarity may have had an effect in this study. 

The IRS Free File program is not used in a vacuum; relevant software, as well as contexts of 
prior technology use and other aspects of the user’s life, may impact the user experience and 
public opinion of the program. 


5.1.3 Usability Comparisons by Participant Education 

Table 50 shows the size of each group. The “lower than bachelor’s” group contains participants 
who had earned a high school diploma, GED, or equivalent, and participants who earned an 
associate’s degree. The “higher than bachelor’s” group includes participants who had earned a 
master’s degree and those with a doctorate degree. 
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Table 50. Participant Education Groups Sample Sizes. 



Lower than 

Bachelor’s 

Higher than 


Bachelor’s 

Degree 

Bachelor’s 

Number of 
participants 

6 

13 

10 


Average SEQ Ratings across Participant Education 
Level by Task 

7 



Task 1 Task 2 Task 3 


above average SEQ, >5.6 

= average SEQ, 5.3-5.6 

below average SEQ, <5.3 

• Below Bachelor's Degree 

• Bachelor’s Deg 

• Above Bachelor's Degree 


Figure 25. Average SEQ Ratings Across Participant Education Level by Task. 
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Average UMUX Scores by Participant Education 
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Figure 26. Average UMUX Scores by Participant Education Level. 


Participants in all education levels thought Task 3 was more difficult than Tasks 1 and 2, as 
shown in Figure 25. Users with below a bachelor’s degree reported lower ease on Tasks 1 and 2 
than the other participant groups, and were the only group to rate Task 1 ease of use in the 
“average” response band rather than the “above average” band. Task 3 ease of use diverged 
much more sharply from Task 1 and 2 ease for participants in the bachelor’s and above 
bachelor’s group than it did for users in the below bachelor’s group. For the IRS web pages (e.g., 
Tasks 1 and 2, program discovery and Free File software offer selection), these patterns suggest 
an opportunity to focus usability resources on improving the experiences of taxpayers with 
education below a bachelor’s degree. This group may have unique characteristics that affect their 
internet and technology usage and impact their access to the Free File program. 

However, UMUX scores (overall experience) presented in Figure 26 landed in the acceptable 
range for all education groups. The below bachelor’s group actually rated their overall 
experience highest at an “A”; all other groups gave a “B.” MITRE cannot explain the disparity 
between SEQ and UMUX results, but results could have been affected by differently sized 
groups (six below bachelor’s, 13 bachelor’s, 10 above bachelor’s). There could have been an 
impact from the difference between rating to reflect on a specific task versus thinking back over 
a whole experience. Lastly, different participant expectations going in about the ease or difficulty 
of filing taxes and tax-related activities could have had an effect, i.e., a person who expects taxes 
to be painful and difficult may rate a middling user experience higher than would a person who 
expects taxes to be easy. 
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More research is needed to explore these findings, including further usability testing and iterative 
design processes to improve usability. Given the differences between software provider portion 
SEQ responses and IRS portion SEQ responses, the IRS should encourage Free File partners to 
dedicate resources to improving the usability of their Free File offerings. 

5.1.4 Usability Comparisons by Participant Age 

Participants were grouped into age terciles. Table 51 shows the size of each sample. 


Table 51. Participant Age Groups Sample Sizes, by Age Tercile. 



20-35 years 

36-51 years 

52-85 years 

Number of 
participants 

10 

10 

9 


Average SEQ Ratings across Participant Age by Task 



Task 1 Task 2 Task 3 


above average SEQ, >5.6 
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Figure 27. Average SEQ Ratings Across Participant Age by Task. 
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Average UMUX Scores by Participant Age 
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Figure 28. Average UMUX Scores by Participant Age. 

All age groups reported above-average ease of use for Tasks 1 and 2, and lower ease for Task 3 
(see Figure 27). The youngest group rated Task 3 ease within the “average” band, while all other 
participants thoughts Task 3 had “below average” ease. Figure 28 shows that the youngest user 
group also rated their overall experience the highest, giving it an “A,” while the other groups 
deemed overall experience a “B.” No clear age-usability trend emerges, but there is some 
evidence that the youngest group found the system more usable than other participants did. 

These findings suggest focusing further usability efforts on the middle and older age groups. 

MITRE recommends performing further research to explore these findings, including usability 
testing and iterative design processes. Again, considering the usability differences between the 
Free File software offer task (Task 3) and Tasks 1 and 2, MITRE suggests that Free File partners 
are encouraged to improve the usability of their Free File offerings. 
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6 Debriefing Findings 

This section presents the analysis of taxpayer responses to open-ended questions 
following the completion of testing. Some of the key findings from this section 
include: 

• Nearly one-third of participants stated that they had knowledge of Free File for 
at least the last year. 

• The main thing that taxpayers knew about the Free File was that it was free for 
taxpayers, although four participants conflated FFA members’ commercial free 
filing services with the IRS Free File Program. 

• Most taxpayers said they would turn to internal help options provided within 
the Free File software or would conduct an online search for help. 


After completing all their assigned tasks, participants completed a user experience questionnaire 
to gather impressions on their experience using the IRS Free File Program. 

Additionally, participants answered a set of open-ended debriefing questions. The MITRE team 
conducted content analysis to identify common themes across participants’ responses to six main 
questions: 

• What, if anything, did you know about the Free File Program prior to today’s session? 

• Have you used the Free File Program previously? 

• If you were completing the online tax return by yourself and couldn’t figure something 
out, what would you normally do? 

• Is there anything that could be done to make the process of filling out the form easier? 

• Is there anything that could be done to make using the IRS Free File program easier? 

• If you were a member of the military, what would you think about this site? 

6.1 Overall Experience 

While seven participants stated that they found the overall Free File program easy to figure out, 
four participants did not find it easy to use, with two participants specifically stating that the 
rules for filling out certain elements of the forms were unclear and hard to follow. Specific 
problems that participants mentioned included the Free File software being different from 
previous software, having to go back and forth between documentation and the Free File 
software to enter data, and not being able to find the link on the website. Two participants did 
feel that the layout was well done. 


Table 52. Overall Debriefing Comments. 


Overall Debriefing Comments 

# Mentions 

Easy to Figure Out 

7 

Good Layout 

2 


Not Easy to Figure Out 

2 
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Rules Were Unclear 

2 

Different From Previous Software 

2 

Couldn’t Find Link 

1 


6.2 Previous Knowledge of IRS Free File Program 

Almost a third of the participants (eight of twenty-nine) stated they had knowledge of the IRS 
Free File program for more than the last year. Four participants stated that they had no previous 
knowledge of the IRS Free File program 
prior to last year. The main thing that 
people knew about the program was that it 
was free for taxpayers, although four 
participants also conflated larger company 
free filing services with the IRS Free File 
program. 

Participants had heard about the program 
primarily from three sources: 
advertisements (five participants), heard 
from friend or family members (five 
participants), and online searches (five 
participants) such as Googling free tax 
filing or seeing the program on the IRS 
website itself. 


Quotes: Previous knowledge of Free File 
Program... 

“I looked online probably to find where you could 
print out the forms and that’s how I saw a notice 
about Free File.” 

“I have used it before. I knew it was a portal to 
other business that do tax preparation where you 
can file for free. I know TaxAct and H&R Block 
were a part of it.” 

“I didn’t know that you could do that through the 
IRS Free File.” 


Figure 29. Previous Knowledge Quotes About Free File 
Program. 


Table 53. Previous Knowledge of Free File Program. 


What, if anything, did you know about the Free File 
Program prior to today’s session? 

# Mentions 

Known about the program for a while 

8 

Didn’t know about the program before last year 

5 


Where participants learned about Free File program: 

Looked online for free filing 

5 

Told about program by someone else 

5 

Heard or saw advertisements about program 

5 

IRS website 

2 

Marketplace podcast 

1 

Newspaper 

1 

Email 

1 

VITA Program 

1 
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6.3 Previous Experience Using the IRS Free File Program 

Almost half (14) of the participants felt the Free File program was easy or fairly easy to use, 
although only four of those participants stated that they had used the program prior to the 
usability testing. Two participants also stated that they had used third-party software prior to the 
usability testing and found those software packages easy to use. Four participants also 
specifically commented that they found the experience within the usability testing environment 
easier than when they had done their own taxes previously (two used the IRS Free File program 
previously, and two used third-party software previously). 

Five participants also commented that third-party software and the IRS Free File program 
experience were all similar to one another. Participants also mentioned that there were some 
ways in which the third-party software was easier to use than the Free File software, especially 
because the third-party software carries over taxpayer information from year to year (two 
mentions) and that it was easier to use (one mention). However, two participants mentioned 
being charged by third-party software when they were actually eligible for free tax filing. One 
other difficulty specifically mentioned by a participant was that the rules for eligibility, and 
potentially one’s own tax situation, change from year to year, making it difficult to use the IRS 
program since you cannot maintain the same filing approach every year. 


Table 54. Experience Using Free File Previously. 


Experience Using Free File Previously 

# Mentions 

Fairly easy 

8 

Very easy 

6 


Used Free File before 

8 

Used third-party software before 

5 

All software is similar 

5 


Advantages of third-party software 

Carries over information from prior years 

2 

Eligibility for Free File changes 

1 

Easier to use 

1 


Participants were also specifically asked which third-party software they had used in the past. 

The majority of participants had used Member 10 or Member 2 for Free File in the past (14 and 9 
participants). 


Table 55. Software Used Previously by Participants. 


Software Used Previously by Participants 

# Mentions 

Member 10 

14 

Member 2 

9 

Member 8 

3 

Member 3 

3 

Member 9 

1 
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Member 1 

1 

Member 6 

1 

Member 11 

1 

Don't Remember 

1 

Other 

1 


6.4 Where Taxpayers Would Go for Help 

Most participants stated that if they encountered a situation where they could not figure out what 
to enter on the tax form, they would look in the internal help files for the Free File software, with 
a number of participants specifically mentioning help buttons within the software or an FAQ or 
similar file (see Table 56). Also mentioned (often by the same participants who said they would 
use the help files) was doing a Google or other online search for the answer to the question. Six 
participants specifically mentioned going to the IRS website for answers, and three participants 
said they would call the IRS for help (although one participant also specifically stated that he 
would never call the IRS as they have not been able to answer his questions in the past). 

Participants also mentioned that they would reach out to a friend or family member for help (five 
participants) or would go to a professional tax preparer to complete their taxes if they could not 
figure it out for themselves (eight participants). 


Table 56. Mentions of Where to Go for Help. 


Where to Go for Help with Tax Questions 

# Mentions 

Search for answers on their own: 

Internal help files 

13 

Online search 

13 

IRS website 

6 


Reach out to others for help: 

Professional tax preparer 

8 

Friends or family 

5 

Call IRS 

3 

Contact software 

2 


6.5 Taxpayer Reflections on Process to Complete Return 

Five participants stated that the entire process of completing the Free File return was easy and 
there was nothing they could think to improve the process, while two participants simply stated 
they had no thoughts about improving the process. Of the remaining participants five participants 
felt the biggest issue was that there were too many options to choose from on the IRS website, 
and four participants felt that the lookup tool could make the free options easier to spot or placed 
in a more prominent location on the website. 
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Many participants (seven total) talked about improving the overall layout and design of the Free 
File tool, especially the lookup tool. Three participants stated they would like to make it easier to 
see the form by improving font and readability and potentially font size. Three participants 
wanted to have a better understanding of how the software package was progressing through 
their return, ideally with a map of some sort labeled at the beginning of the process. Finally, two 
participants discussed general improvements in graphic design, such as higher contrast colors 
and identifying what specific areas in the lookup tool are for. 

Two participants also discussed the need for the third-party tools to make it clear that taxpayers 
are eligible to file for free (one participant stated he was previously charged even though he was 
eligible for free filing), and to make it clear through IRS communications that free filing only 
goes through the IRS website to avoid being charged. One participant suggested that Free File 
should be available to all taxpayers regardless of income. 

When asked about how to make the overall process easier, participants stated that they would 
like if the IRS website could suggest a Free File software offer for them rather than having to 
choose, that the IRS website would use more plain language, that they would like to be able to 
import their data into the Free File tool rather than having to manually enter it, and that they 
would like to have more training on tax preparation. 


Table 57. Improve Filing Process Through Free File. 


How to Improve the Process of Filing through Free File 

# Mentions 

Issues with the process: 

None 

5 

Too many options to choose from 

5 

Make free software easier to identify 

4 

Clear eligibility 

3 

Free File only goes through IRS 

2 


Suggested changes: 

Better layout 

7 

Improve font and readability 

3 

Improve mapping of software 

3 

Improve graphic design 

2 

Make free file available to everyone 

1 


Improve process: 

Suggest a vendor 

1 

Plain language 

1 

Import data 

1 

More training on tax preparation 

1 
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6.6 Taxpayer Reflections on Support from the IRS 

As with improving the process, participants primarily discussed improving the graphic design of 
the Free File site and lookup tool (three participants). Participants also suggested advertising the 
program better, recommending a Free File software offer to taxpayers, and having clearer 
language regarding eligibility and the process. Specific to the process of navigating the website, 
one participant repeated that they were charged by third-party providers when they were eligible 
for free filing, wanted to be able to exit a particular software provider without having to save 
their information or to register without having to provide a personal email. 


Table 58. How the IRS Can Help Taxpayers With Free File. 


What Can the IRS Do To Help? 

# Mentions 

Graphic design 

3 

Advertise 

3 

Recommend a vendor 

2 

Clear description of eligibility 

2 

Clear explanations of terms 

2 

Exit without saving 

1 

Register without an email 

1 

Charged even though eligible 

1 


6.7 Taxpayer Reflections on Entering Military Pay 

Although no participants in the study had received military pay, each participant was asked what 
they thought they would do if they were a member of the military. One participant stated flatly 
that they had never served and could not answer the question. Thirteen participants stated that 
they would simply check the box on the lookup tool for receiving military pay and five 
participants stared they would use the specific military link on the lookup tool. 

Six participants stated they would look for military options within the lookup tool. Three 
participants stated they didn’t see anything specific to the military in the lookup tool, and two 
believed that the military link should be made clearer on the website. 

Five participants believed that military members would have access to information specific to 
their filing responsibilities. Three participants stated they would use military specific resources 
to learn about how to use Free File, while two participants simply stated that military members 
would know how to use the Free File software. 


Table 59. Actions With Receiving Military Pay. 


What would you do if you received military pay? 

# Mentions 


Use lookup tool specific options: 

Check box for military pay 


13 

Use military link on website 


6 


Look for possible options within lookup tool: 
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Look for military options 6 

Didn’t see military option 3 

Make military link more clear 2 


Specific military knowledge 

Use military-specific resources 3 

Military members would know how to use Free File 2 


79 

PRE-DECISIONAL - FOR INTERNAL IRS W&l USE ONLY - NOT FOR PUBLIC RELEASE 






PRE-DECISIONAL - FOR INTERNAL IRS W&l USE ONLY - NOT FOR PUBLIC RELEASE 


7 Taxpayer Pathways and Metrics 

This section gives a summary of MITRE’s findings from Taxpayer Pathway data 
collected in Tasks 1 and 2 of the taxpayer testing sessions. A user pathway is a series of 
steps the user performs to complete a task during an interaction with a system of interest. 
Participant pathways show how many participants succeeded or failed tasks, what routes 
led them to frustration, inefficiency, or errors, and what user decisions triggered various 
pathways. This section also describes the Free File software offers that taxpayers selected 
and their strategies for selecting an offer across all tasks. 

Some key findings from this section include: 

• A majority of the search terms test participants used to find the Free File 
landing page included the term “IRS,” and nearly half used search terms that 
included “IRS” and “free.” However, none of the search terms used by the 
general population to find the Free File landing page included the term “IRS.” 

• Twenty-four taxpayers found the Free File landing page as a part of Task 1; 14 
taking an optimal path and 10 taking a sub-optimal path. 

• The only error taxpayers committed in the Lookup Tool was incorrectly 
selecting “yes” to EITC question. 

• Across all tasks, the most co mm on strategy taxpayers used to select a Free File 
Software Offer was choosing based on past experience with a software 
provider, followed by familiarity with the product name or URL. 


This section discusses the user flows, or pathways, participants followed during Tasks 1 and 2, 
for example, the series of web pages the user visited and actions they took at each step. 
Participant pathways were examined and grouped into similar routes to identify co mm on 
pathways and unveil points at which users left optimal paths. Each path is categorized as one of 
the following: 

• Acceptable: Any route that ultimately allows the user to complete the task 

• Unacceptable: Any route that does not ultimately allow the user to complete the task 

Task completion is defined as: 

• Task 1: Locating the Free File landing page. The task started from the computer desktop 
with no browsers or windows open. 

• Task 2: Selecting a Free File software offering. The task started from the Free File 
landing page. 

Acceptable paths are further divided into: 

• Acceptable-Optimal: Acceptable paths that do not require the user to backtrack, do not 
cause the user frustration (frustration is measured through user comments), and in which 
the user does not commit errors 

• Acceptable-Suboptimal: Acceptable paths in which users backtrack, voice frustration, 
or commit errors 
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7.1 Task 1: Free File Landing Page Findability Testing 

In this section, MITRE reports on the user flows that taxpayers followed for the first task in the 
usability test, finding Free File and navigating the Free File landing page. The interviewers 
allowed taxpayers to move forward with the task up until the point of selecting a Free File 
software offer for themselves in Task 1. User flows fell into the following seven general 
pathways, displayed in Table 60. 


Table 60. Taxpayer Pathway Types from User Flow Analysis (Task 1). 


Path 

Path Result 

Pathways Including Visiting IRS.gov 

Path 0: Navigate to IRS.gov; never arrive at Free File landing page 

Unacceptable 

Path 1: Navigate to a page on IRS.gov that was not the Free File landing page; then navigate 
IRS.gov site to arrive at the Free File landing page 

Acceptable - Optimal 
Acceptable - Suboptimal 

Path la: Internet search that led directly to IRS.gov Free File landing page 

Acceptable - Optimal 

Path lb: Internet search to land directly on Free File landing page; then taxpayer got distracted or 
lost on other areas ofIRS.gov 

Acceptable - Suboptimal 

Pathways Not Including Visiting IRS.gov 

Path 2: Select commercial option(s) from search results and end there 

Unacceptable 

Path 2a: Select commercial option(s) from search results, back up, and select IRS option from 
search results 

Unacceptable 

Path 3: Navigate directly to software provider website (no search first) 

Unacceptable 


The results of paths 0, 2, and 3 did not lead to task completion. Paths 1 and la could have been 
acceptable-optimal and paths 1,1b, and 2a could have been acceptable-suboptimal, depending on 
the taxpayers’ experience following the path. The difference between a path falling in 
acceptable-optimal or acceptable-suboptimal was due to the participant expressing frustration or 
confusion with the experience. Two participants first followed unacceptable paths and then 
acceptable/not optimal paths after interviewer prompting. These participants are thus recorded as 
following unacceptable routes in the path counts. Figure 30, Figure 31, and Figure 32 provide 
graphical representation of the pathways with comments from participants’ experience 
completing the task. 
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I'm "familiar with it and used it in past to 
get information on refunds...assume most 
people would either go to IRS.gov or know 
to search for IRS in a search engine" P6 


Lands on 
IRS.gov 


home 



Navigate 


Lands on Free 
File Landing 
Page 


Lands on Free 
File Software 
Offers 


freelHe 


freeflie ) 


Google search 


Types in "IRS" 



Task 1 Path la 


Types in "IRS free tax filing" 


"...a bunch of different options pop up; I 
would personally go to this one here with 
the IRS, not the ads, and click here to see 
what happens." PI 3 


Figure 30. Task 1 Acceptable-Optimal Pathways. 



“This is very confusing, I'm not sure how I got 
here...This [Free File link in menu, and the Free 
File button on the page] is under 'popular 1 ? 
Seems to me it should be more prominent." P5 


Lands on 
IRS.gov 
home 


Navigates 

IRS.gov 


SIRS SIRS 



PI 9 selected "About the Free File 
Program" while attempting to 
locate the list of software offers. 



Types in "US IRS" 




P28 selected e-file.com because it was 
the first result on the list. They navigated 
to the create your account page before 
returning to Google. 


Types in "IRS free file" 


Lands on 
commercial 




PI4 navigated away from the 
Free File Landing Page and 
eventually entered their zip and 
state to review Authorized IRS 
e-File Providers..."Oh wait. 
There's people's names...This is 
confusing. I don't know what 
this means...[What were you 
looking for?]...l'm looking for 
authorized providers....[but] I 
don't want a person because I'd 
have to pay something." 



Figure 31. Task 1 Acceptable-Suboptimal Pathways. 
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Lands on 
IRS.gov home 



Navigates 

IRS.gov 


Selects How to file, then Find a tax pro now. "I find this 
overwhelming. On my own I would play around and look 
through the authorized IRS e-file Providers. I'm looking for a 
tax form because I wanted to fill it out myself" P3 


“ Task Failure: User gives up 
k search 


Types in "IRS Free File" 


Google search 


Task 1 Path 0 


Task 1 Path 3 


Selects first link in Bing results 


Types in URL 


Types in "Free File USA" 


Lands on 
commercial 
site 


Lands on 
commercial site 

Vendor I 


"Task Failure: User selected 
k commercial site 


"File in 15 minutes or less, 
that sounds great" P4 


Logs in to 
commercial site 




Task Failure: User selected 
commercial site 



Notes the website 
charges for state; 
doesn't know what to 
do next. Thought the 
website looked like it 
was from IRS.gov. "I 
was just looking for 
something that said 
free tax...I was looking 
for something that 
was an official site, 
the top one just looks 
like an ad so the 
second one looks like 
the right one...when I 
went to the website I 
thought it was right, 
but as I looked it may 
not be" P4 


'I have login information, I would 
use that, it will prompt me to start 
filing my taxes.” PI 5 

Figure 32. Task 1 Unacceptable Pathways. 


Table 61. Task 1 Path Rating Scale. 


Acceptable - Optimal 

Acceptable - Suboptimal 

Unacceptable 

Path 1 Path la 

Path 1 Path lb Path 2a 

Path 1 Path 2 Path 3 

Number of 3 11 

3 2 5 

1 2 2 

participants 14 

10 

5 


Twenty-four of 29 (82.8 percent) participants were able to find the Free File landing page. Of 
these, slightly more followed optimal routes (14 optimal, 10 suboptimal). Suboptimal routes 
typically involved participants getting lost within the IRS website, such as not being to find the 
link to the Free File landing page from the home page (see usability issue in 4.2.3.1). Future 
usability efforts should focus on increasing usability specifically at the points where users fell 
away from the optimal path, explored further in Section 4.2 and in the taxpayer journey. 

An equal number of participants experienced optimal and suboptimal routes when starting from 
IRS.gov, suggesting an opportunity to change the IRS public web page design to better support 
navigation. 

In general, improving the design of commercial tax software providers’ pages is outside of the 
scope of this independent assessment. However, participants shared information about the key 
things they assessed when visiting Free File software providers websites. This information can 
help the IRS and Free File software providers understand the taxpayer experience when 
evaluating their tax preparation options. When participants visited commercial pages, they said 
they were looking for things like: 

• Fees 
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o To avoid: Mention of any fees or charges 
o To seek: The word “free” 

• Security 

o To avoid: “.com” ending to the URL 

o To seek: Visual design that “looked legitimate,” “not like a scam,” and/or 
“seemed reliable” 

o To seek: “.gov” ending to the URL 

o To seek: Something that looked “like it is officially government” and/or “part of 
the IRS” 

o To seek: Something that looked or sounded familiar 

• Relevance 

o To seek: Eligibility requirements 

7.1.1 Search Terms for Locating Free File 

MITRE captured the search engine query terms that taxpayers used to seek where they would file 
their federal return for free. 

Almost all participants began with a search engine query, suggesting that the IRS should 
consider search engine queries as a starting point to many taxpayer-IRS interactions. This means 
the IRS should either be prepared for users to enter IRS page navigation flows from unexpected 
points or optimize for search engine queries and results in order to guide taxpayers toward 
specific starting points. 

One limitation of this study is that participants may have been primed to look specifically for 
IRS Free File when they would not have done so otherwise. The recruitment materials, including 
the letters sent by the IRS and MITRE and the recruitment website, mentioned the Free File 
program by name, possibly leaving participants with an initial impression that this language 
choice was appropriate during their participation in the study. Additionally, the consent form and 
interview protocol used the term “IRS Free File program” in the introduction when the 
interviewer explained the purpose of the research. Once the usability part of the task began, the 
protocol did not mention IRS Free File by name, and instead asked participants to “imagine they 
learned that the IRS offered a program to file their federal tax return for free” (see 9Appendix C 
for recruitment and interview materials). Given the repeated use of “IRS Free File” and “IRS 
Free File online software program,” MITRE may have given participants familiarity with these 
terms that they would not have had otherwise. 

The search engine analysis using Google Analytics reported in the Free File Program 
Assessment Final Report provides a full picture of taxpayer behavior that leads to the Free File 
landing page. In this section, MITRE reports the search terms used by the usability test 
participants and reflects on how the format of the interview may have introduced bias by priming 
participants to think about IRS Free File in those terms. Figure 33, Figure 34, and Figure 35 
provide graphical representation of the pathways with comments from participants’ experience 
completing the task. 
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Reviews offering descriptions below the look 
up tool. Read through most offerings, even 
after finding offerings they thought would 
work. "I have a lot of choices. My gut is too 
many choices, because I would like explore 
each one of them. I would like to explore each 
one of them and see how they would guide 
through the options." P10 



Lands on Free File 
Software Offers 



k 2 Path i Reads list Reads eligibility criteria 

of vendors on list of vendors 



Lands on Free 
File Lookup Tool 



Lands on 
commercial site 



"...[I] would choose a vendor 
[I] have used before like H&R 
Block or TurboTax." P3 

Figure 33. Task 2 Acceptable-Optimal Pathways. 



Lands on Free File 
Software Offers 



Selects the first offer “I like to 
choose the first one, I usually feel 
the first one is the best one." Sees 
Illinois is not eligible, returns to 
software offers and selects the next 



Figure 34. Task 2 Acceptable-Suboptimal Pathways. 
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"I'm not 100% sure where 
to go from here so I'd 
click this big blue button. 
It's to see if I'm eligible so 
it's not like TurboTax 
where you hit start." P29 


Reads list Reads eligibility criteria 

of vendors on list of vendors 


Lands on Free File 
Software Offers 



Task 2 Path 2 


P4 selected "Individuals” 
and read the "What you 
need to know” section, 
then selected Form 1040, 
then "current year." 




mm s 


Navigates 

IRS.gov 


Lands on Free 
File Lookup Tool 



Figure 35. Task 2 Unacceptable Pathways. 


Selects vendor 



Lands on 
commercial 
site 



Table 62 shows the top 10 search terms that drove organic web traffic to the Free File landing 
page on IRS.gov, according to the separate web analytics research conducted by MITRE (see the 
Free File Program Assessment Final Report for details). Table 63 displays all unique search 
terms used by participants in the usability study for Task 1 (finding where to file their federal 
return for free), with the number of participants that used those exact keywords and the result of 
using those keywords in terms of success (whether they found the Free File landing page) and 
whether their path to the Free File landing page was optimal or suboptimal. Table 63 does not 
include URLs that participants directly typed into the address bar. 13 

A key characteristic of all 10 of the search terms used by the general population to arrive at the 
Free File landing page reported in Table 62 is that they do not contain the term “IRS.” In 
contrast, MITRE found that the majority of search terms used by participants in the usability 
study included the term “IRS” (Table 63). This could be attributed to the repeated exposure to 
the IRS’s affiliation with the study. Seven unique search terms used by 14 participants including 
both the term “IRS” and “free,” and the most commonly used search term among MITRE 
participants was “IRS free file” (n=6). Notably, the search terms that combined these keywords 
appear to have driven the most successful internet searching behavior in MITRE’s usability 
study. In comparison, the search terms that are most commonly used in the general population, 
which do not contain reference to the IRS (e.g. “free tax filing”), led to the selection of 
commercial options among MITRE’s usability participants. 

These results suggest that exposing taxpayers to the singular concept of “IRS Free File” may 
help taxpayers more successfully locate Free File services in online searches and avoid 


13 One taxpayer typed “irs.com” into the address bar and did not realize her own mistake, landing on a commercial 
vendor site. Another went straight to turbotax.intuit.com from the address bar. Two taxpayers navigated directly to 
IRS.gov from the address bar and were successful in finding the Free File landing page, though they followed 
suboptimal paths. 
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commercial options. These results also suggest that taxpayers in the MITRE usability study were 
likely biased by their exposure to the combination of “IRS” and ‘Tree File” or “free tax filing” in 
the course of the study, and this may have affected their behavior in the interview. Thus, 
taxpayer online searching behavior in the usability study may not be generalizable to the broader 
taxpayer population. 

Table 62. Top 10 Search Terms Driving Internet Traffic to Free file Landing Page (From Web 

Analytics Analysis). 


Term 

Rank 

free tax filing 

i 

free online tax filing 

2 

free taxes online 

3 

free file 

4 

free e file 

5 

free federal tax filing 

6 

free income tax filing 

7 

free tax return 

8 

file federal taxes 

9 

free tax return filing 

10 
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Table 63. Search Terms Used by MITRE Study Participants in Task 1. 


Term 

Count 


Result 

Search terms including ”IRS” but not ’’Free” 

US IRS 


1 

Successful (suboptimal path) 

IRS 


1 

Successful (suboptimal path) 

IRS.gov 


1 

Successful (suboptimal path) 

IRS file online 


1 

Successful (suboptimal path) 

IRS 1040 form 


1 

Successful (suboptimal path) 

Search terms including ”IRS” and ’’Free” 

IRS free file 


6 

3 successful (optimal path) 

2 successful (suboptimal path) 

1 not successful; ended on IRS Fillable Forms 

IRS free filing program 


1 

Successful (optimal path) 

free file software IRS 


1 

Successful (optimal path) 

IRS free tax filing 


3 

Successful (optimal path) 

file taxes free irs 


1 

Successful (optimal path) 

free tax filing federal irs 1040 


1 

Successful (optimal path) 

free file software IRS 


1 

Successful (optimal path) 


Search_terms not in cludi ng ’’IRS” 

1 successful (optimal path) 


file taxes for free 

2 

1 successful (suboptimal path) 

free file USA 

1 

Not successful; ended on freetaxusa.com 

free file 

1 

Not successful; ended on e-file.com 

free tax filing 

1 

Not successful; ended on freetaxusa.com 


7.2 Task 2: Lookup Tool Testing 

MITRE captured the user flows that participants exhibited when they first encountered the Free 
File landing page and were asked to select a Free File software offer. User routes followed four 
general pathways, shown in Table 64. 


Table 64. Taxpayer Pathways Followed to Select a Vendor (Task 2). 


Path 

Path Result(s) 

Path 1. Uses Free File offerings descriptions to find a Free File software offer (does not use 

Acceptable - Optimal 
Acceptable - Suboptimal 

lookup tool) 

Unacceptable 

Path la. Uses Free File offerings descriptions to find a Free File software offer (does not use 
lookup tool); backtracks to choose a different Free File software offer after selecting initial 
Free File software offer and reviewing the software provider website 

Acceptable - Suboptimal 

Path 2. Uses lookup tool to find a Free File software offer 

Acceptable - Optimal 

Path 3. Unable to locate software offerings page from Free File landing page 

Acceptable - Optimal 
Acceptable - Suboptimal 
Unacceptable 
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Path 3 did not lead to task completion, and there were cases where paths 1 and 2 did not either. 
Paths 1 and 2 may be acceptable-optimal and paths 1,1a, and 2a may be acceptable-suboptimal. 

One participant is excluded from Task 2 pathways analysis because they were prompted by the 
interviewer to use the lookup tool when their first scenario was described. 


Table 65. Task 2 Path Rating Scale. 


Acceptable - Optimal 

Acceptable - Suboptimal 

Unacceptable 

Path 1 Path 2 

Path 1 Path la Path 2 

Path 1 Path 2 Path 3 

Number of 7 10 

1 1 5 

1 2 1 

participants 17 

7 

4 

Out of total 

participants 17/28 = 60.7% 

7/28 = 25% 

4/28 = 14.3% 


Table 66 describes how many participants elected to use the lookup tool versus the Free File 
software offer descriptions when selecting a Free File software offer for the first scenario issued. 
In this scenario, the interviewer did not prompt the participant to use the lookup tool; they were 
permitted to select a Free File software offer in whatever way was most natural for them. 

Table 66. Use of Lookup Tool versus Vendor Descriptions for Scenario 1. 


Accep table Paths 



Used Offerings Descriptions 

Used Lookup Tool 


(Paths 1, la) 

(Path 2) 

Number of participants 

10 

17 

Out of total participants 

10/28 = 35.7% 

17/28 = 60.7% 


More participants relied on the lookup tool (17 users) than on the Free File software offer 
descriptions (10 users) to select a Free File software offer. However, many users did use the Free 
File software offer descriptions instead. As long as both options are available to taxpayers, 
usability issues in both will need to be addressed since many taxpayers will be using both. 

Section 4.2 describes usability issues that led participants away from the lookup tool, including 
unfamiliar naming, an unclear description, dense text blocks, and visual element appearance. If 
the IRS wants to direct taxpayers toward the lookup tool, these usability issues need to be 
addressed to make the tool more findable. 

Optimal paths were followed by 60.7 percent of participants. Section 4.2 details usability issues 
whose solutions could help move suboptimal and unacceptable path users toward optimal paths, 
specifically, areas where taxpayers were confused or frustrated, committed errors, or had to 
move backwards through web pages. 

7.2.1 Errors Committed Using the Lookup Tool 

Incorrect answers to EITC eligibility were the only errors committed during lookup tool usage. 
See Figure 36 and Figure 37 for the results returned from the two selection scenarios when the 
scenarios were completed correctly. Table 67 shows errors committed using the lookup tool for 
selection scenario 2, which is the only task where all participants were required to use the lookup 
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tool to select a Free File software offer. Eight of 28 participants committed an error on the 
lookup tool on this task; all errors were due to selecting yes for EITC eligibility. 14 

Various lookup tool usability issues—unclear eligibility information, lack of clarity of the 
consequences of wrong answers, and lack of verifying taxpayer-entered information for 
correctness—could be solved by instead asking the taxpayer about information that determines 
EITC eligibility, like income (already asked), anticipated filing status, and number of children. 
Section 4.2 captures further detail and alternative suggested solutions. 

Table 67. Errors Committed using Lookup Tool (Scenario 2). 


Error 

Count 

Entered "Yes" for EITC question 

8 

No Error 

20 

Total 

28 


7.3 Participants’ Free File Offer Selections Across All Tasks 

Throughout the usability testing tasks, participants were presented with multiple opportunities to 
select a Free File software offer (using the lookup tool or the eligibility criteria on the software 
offers page), and participants shared their rationale for selecting the Free File software offers 
they chose. 

Table 68 shows the Free File software offers selected across all usability tasks in the session 
where taxpayers had the opportunity to select one. 


Table 68. Free File Offers Selected During the Usability Session. 


Free File Software Offer 

Count 

Percent of Total 

Member 10 

31 

32% 

Member 3 

14 

15% 

No Offer Selected 

13 

14% 

Member 2 

11 

11% 

Member 8 

5 

5% 

Member 4 

5 

5% 

Member 1 

4 

4% 

Member 6 

4 

4% 

Member 7 

3 

3% 

Member 5 

2 

2% 

Member 11 

2 

2% 

Member 9 

1 

1% 

Member 12 

1 

1% 

Total 

96 

100% 


14 One taxpayer, not included in the n=28, was frustrated by not seeing Texas in the lookup tool state drop down and 
opted to not use the lookup tool for selection scenario 2. 
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Member 10 was selected the most often (32 percent of selections), followed by Member 3. 
Participants did not select a Free File software offer or said they would select multiple 14 percent 
of the time. 

Participant comments made throughout the sessions illuminate how taxpayers understand the 
Free File software offers and the strategies they use to make a selection when confronted with 
many options. Table 69 shows the primary reasons taxpayers gave for selecting software offers, 
among all taxpayers who selected a Free File software offer at any point during the interview 
(N=83). 

Table 69. Reasons for Software Offer Selections. 


Reason 

Count 

Prior experience with Free File software provider 

27 

Familiarity with the product name or URL 

18 

Based on Free File member’s eligibility criteria 
listed on software offers page 

13 

Picked at random 

7 

First offering listed on lookup tool results or 
software offers page 

6 

Unknown/not mentioned 

6 

Searched for terms in Free File software offer 
provider’s name (e.g. ’'free,” "file," "IRS") 

5 

Reviewed website of each offer listed in lookup 
tool results 

1 

Total 

83 


The most common strategy taxpayers relied upon to select a Free File software offer was past 
experience with a software provider (n=27). If they had had a positive experience with the 
software provider in the past, they felt confident that the software provider offering would work 
well for them or the fictional “Isaac” or “Rita.” Both positive and neutral past experiences were 
cited as reasons to use a software provider again, specifically for efficiency—users said their 
personal information would be saved and thus shorten the filing process both at present and in 
the future. 

When the top name recognition offerings, were available, users usually chose one of the two due 
to familiarity with the name (n=18). This was the second most commonly stated reason for 
choosing an offer. Name recognition usually came up as a positive aspect (selecting a software 
provider they recognized) but a few times as a negative (wanting to avoid “the big guys”). 

The next most co mm on strategy for selecting a Free File software offer was based on the Free 
File offer’s eligibility criteria listed on the Free File software offers page (n=13). Next, 
participants chose at random (n=7). 

Following selecting at random, participants choose an offer because it was first in the list (n=6); 
this was especially in cases where the taxpayer was not familiar with any of the other names. 
One stated, “I chose this vendor because I didn’t recognize any of the three and it was the first 
one listed.” This suggests that the practice of randomizing the Free File software offers is 
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effective for distributing taxpayers’ choices across software providers, but only in cases where 
prior experience and name recognition do not exist. 

When they were not familiar with any of the software providers’ names from advertising or prior 
experience, another strategy participants used was to evaluate aspects of the software provider 
name, searching for keywords (n=5). Participants talked about software provider names that 
seemed more or less trustworthy to them, and names they perceived as more relevant to filing tax 
returns for free, like Free 1040 Tax Returns (recognizing “1040” and “free”) and FreeTax USA 
(“free”). One participant stated about choosing 1040.com Free File Edition, “I’ll just pick this 
one.... This is the general tax filing form [the 1040],.. it was a place to start.” Some participants 
suggested that there was an element of random selection combined with heuristic devices like 
name to help them narrow down the many options: “I looked to see if I qualify based on my 
income and the state I live in. The other thing was the clever name [ezTaxRetum.com], I admit 
it’s a stupid reason. There are too many choices. If there are ten there and four qualify, then it’s 
like eenie meenie miney mo.” 

One taxpayer reviewed the website of each offer listed in the lookup tool results. 

Note that the data presented here is across both lookup tool and offering descriptions paths. It 
does not account for cases where participants made errors in using the lookup tool or reading 
offering descriptions. This data does not necessarily reflect what users would choose for 
themselves (they were using fictional scenarios) nor selections that were definitively in the 
correct range of responses for the scenarios—eligibility misunderstandings or mistakes led 
participants to different ranges of possible responses. This is an added variable for lookup tool 
paths in particular; when participants erroneously selected EITC eligibility in the tool, that 
affected the range of possible Free File software offers that the tool delivered. 


SHIRS 


Free File Software Offers 


Free File Software Lookup Tool Results 

Based on the information you provided, you are eligible to use Free File software to file your federal and state taxes for free. You may pick from 
either Free Federal and State Return Options or Free Federal Return Only Options. Review the Free File software options and pick the option 
that’s best for you. To view the list of forms and schedules, customer support, and other services, go to the company’s website for more 
information. 

You qualify for the Free File software options below. 


Free Federal and State Return 
Options: 

• Online Taxes at OLT.com 

• TaxAct “ Free File 

• 1040.com Free File Edition 


Free Federal Return Only Options: 

• FreeTax Returns.com 

• eSmart Free File Edition 

• 1040NQW.NET 

• FileYourTaxes.com 

• ezTaxReturn.com 


Figure 36. Free File Offer Lookup Tool Result for Correct Entry of Selection Scenario 1 

Information. 
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Free File Software Offers 


Free File Software Lookup Tool Results 

Based on the information you provided you are eligible for Free File software. 

You qualify for the Free File software options below. 

Free Federal Return Only Options: 

• TaxAct-Free File 

• H&R Block's Free File 

• eSmart Free File Edition 

• Online Taxes at OLT.com 

• TurboTax Free File Pro g ram 

• TaxSla ver 

• 1040.com Free File Edition 

• FreeTaxUSA®!RS Free File Edition 

• FileYourTaxes.com 


Figure 37. Free File Offer Lookup Tool Result for Correct Entry of Selection Scenario 2 

Information. 
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8 Conclusion 

This independent assessment investigated taxpayers’ experience and satisfaction with the online 
Free File system. Specifically, MITRE assessed: 

• Usability issues and taxpayer experiences locating the Free File program online; selecting 
a Free File software offer using fictious tax scenarios, including with the Free File 
software lookup tool; and filing a return with a Free File software using fictious tax 
scenarios 

• Taxpayers’ qualitative insights on their previous knowledge of Free File, their previous 
experience with Free File, and where they would go for assistance while filing taxes 

• Taxpayers’ overall perceptions of the Free File system and program. 

MITRE conducted testing only with taxpayers who had previous Free File use as indicated by 
the Free File indicator in CDW and participants’ own reporting of Free File use during testing 
eligibility screening. Caution should be used when generalizing from the sample of taxpayers 
interviewed here to the full taxpayer population. 

MITRE observed several high severity usability issues across the three main testing tasks that 
taxpayers performed. These issues may halt or delay taxpayers’ progress filing through Free File. 

MITRE found that taxpayers struggle to navigate through IRS.gov to locate the Free File landing 
page and take sub-optimal routes to find it. Once on the Free File landing page, MITRE found 
that taxpayers experienced several usability issues with the software lookup tool, including 
challenges finding the tool, understanding its purpose, and answering questions about state 
income tax returns and EITC eligibility. Through taxpayers’ use of the lookup tool, MITRE 
found that taxpayers do not understand the EITC eligibility criteria. 

MITRE found that taxpayers are overwhelmed by the number of Free File software choices and 
the information provided with them, which makes it difficult for them to select an offer. 
Taxpayers use various strategies to select an offer. Most taxpayers rely on familiarity with 
company names or previous experience with a software provider rather than eligibility criteria to 
select a Free File software offer. 

Broadly, MITRE found that taxpayers lack an awareness that Free File exists, or they 
misunderstand the program. Taxpayers often confuse Free File offerings with free commercial 
products. 

MITRE recommends that the IRS consider taxpayers’ journey with Free File to begin upon 
opening a search engine as nearly all participants started their search for Free File with an 
internet search engine query. The IRS should conduct future research to understand taxpayers’ 
awareness and understanding of Free File program and to understand their reasons for using or 
not using Free File. 
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Appendix A Detailed Methods 

This section provides additional detailed information about the methodological 
approach for the Free File usability study. 

In this section, the population of interest (Tax Year (TY) 2018 free filers) is described 
in terms of their Adjusted Gross Income (AGI), age, and other characteristics 
observable in Compliance Data Warehouse (CDW). 

The recruitment procedures are described, and this section reports detailed response 
rates from MITRE’s recruitment efforts. 

This section also details the data collection methods and the analytical approach, 
including how the interviews were conducted and the literature and theory behind the 
methodological approach. 

A.1 Population of Interest and Sampling Procedure 

The population of interest for this study is taxpayers who are identified in CDW as having filed 
their returns through the Free File program in TY 2018. Because the usability testing focuses on 
the Free File system, Wage and Investment (W&I) asked that MITRE focus on previous free 
filers as the population most affected by this change. Focusing on this population also ensured 
that usability issues would not be due to including participants who would not normally be 
eligible for Free File or who would not normally file electronically. Note that MITRE was not 
able to verify whether returns marked in the CDW table ETARAS MEF1544 were recognized 
by Free File vendors as such; MITRE relied solely on the data captured in CDW to identify the 
population. 15 Because the data were drawn from tax documents, MITRE did not have access to 
some demographic characteristics associated with the taxpayers, such as education level. 

Table 70 describes the full free filer population in TY 2018. These data were drawn from the 
CDW table ETARAS MEF1544, defined as having used Free File 

(ENGLISH_FREE_FILE_RET_IND=1), having the return accepted (accepted_rtn_ind=l), and 
having non-missing values for software vendor ID (softwareidnumber). To clean the data, 
MITRE removed duplicate Tax Identification Numbers (TIN), TINs with missing zip codes, 
TINs with international zip codes, and taxpayers under 18 years of age. 


15 Note that some taxpayers marked as having used Free File did not appear to be eligible for Free File due to high 
AGI. MITRE cannot determine why this is the case. 
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shows an overview of the full sample selection process. 



•Clean data for 
missing zip 
codes, 

international zip 
codes, etc. 
•Analyze Free 
Filer data from 
CDW for 
distribution of 
AGI, age, etc. 


•Evaluate density 
of Free Filers in 
zip code clusters 
used for MITRE's 
previous 1040 
Form testing 
(clusters in 
Chicago, IL; 
Pittsburgh, PA; 
Dallas, TX; and 
Portland, OR) 


• Select zip code 
clusters from 
candidate clusters 
by evaluating 
Free Filer density 

• Selected cluster 
of six zip codes 
in Chicago, IL 


•Randomly select 
2,500 taxpayers 
from the Chicago 
zip codes to send 
invitation letters 


Figure 38. Overview of Sampling Process. 


The full population of taxpayers that MITRE used to construct an invitation sample to participate 
includes 2,343,235 unique TINs from TY 2018. As documented in , MITRE first cleaned the 
data set drawn from CDW, then examined the characteristics of the national free filer population. 
The characteristics of this population are displayed in Table 70. On average, free filers in the 
U.S. have a mean AGI of $23,247, with mean wage income of $21,108. Their average age is 
about 38 years, and the majority file as single (71 percent). Most Free File users in TY 2018 did 
not claim the EITC; among those who did, the mean value claimed was about $565. Table 78 in 
Appendix D shows the Free File vendors used by the national Free File population, Chicago zip 
codes population, and Chicago invitation letter sample. 

MITRE then evaluated the density of free filers in four zip codes previously visited for a separate 
usability testing project (the 1040 Form redesign study conducted by MITRE in late 2018). 
MITRE determined that Chicago had sufficient volume of free filers (N=4,106). MITRE 
examined the Chicago zip codes population characteristics in comparison to the national Free 
File population. This comparison is shown in Table 77 in Appendix D. The comparison shows 
that the Chicago zip code cluster population is less variable than the national population, with a 
clustered distribution around 30 years of age (Figure 39), and has higher AGI, particularly 
among taxpayers in their late twenties and early thirties (Figure 40). 

MITRE then randomly selected 2,500 of the 4,106 taxpayers in the Chicago zip code cluster to 
receive invitation letters. The characteristics of these 2,500 randomly selected taxpayers 
compared to the 4,106 population in Chicago and the full national population are shown in 
Appendix D Table 77. 
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Table 70. Characteristics of National Free Filing population, TY 2018. 


AGI (Dollars) 

Mean 

23,247 


Median 

20,383 


Wage Income (Dollars) 

Mean 

21,108 


Median 

18,079 


Age (Years) 

Mean 


37.9 

Median 


32.0 

Filing Status (Percentage) 

Head of Household 


17% 

Married Filing Jointly 


11% 

Married Filing Separately 


2% 

Single 


71% 

Widow/er 


0% 

EITC Claims (Dollars) 

Mean 


565.9 

Median 


0.0 

Total N 

2,343,235 




l~l Local l~l 


National 


Figure 39. Distribution of Age for National Free Filers and Local (Chicago) Free Filers, TY 2018. 
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Figure 40. Joint Distribution of Age and AGI among National Free Filers and Local (Chicago) Free 

Filers, TY 2018. 
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A.1.1 Site Selection 

The goal of this study was to gather usability and perception information from a sample of 
taxpayers who are indicated as having used Free File in TY 2018. The study team agreed to 
accomplish this by sending out invitation letters to participate to free filers in one location in the 
continental U.S., selected on the basis of two primary criteria. Because of the quick turnaround 
time on this research project, MITRE looked to prior experience conducting usability testing 
with paper filers in the fall of 2018 for the independent assessment of the redesigned Form 1040. 
In that previous study, MITRE interviewed 73 taxpayers in zip code clusters in Chicago, IL; 
Portland, OR; the Pittsburgh, PA metropolitan area; and the Dallas, TX metropolitan area. 
Because of the team’s prior familiarity with these sites, MITRE examined the feasibility of 
revisiting one of these sites for the Free File usability study. To do this, the lead researcher 
examined the demographic characteristics and density of free filers in the zip code clusters 
previously defined for the Form 1040 study. The final zip codes selected appear in Table 71. 
Final Zip Codes Selected for Invitation Sample. 


Table 71. Final Zip Codes Selected for Invitation Sample. 


Area 

Zip Codes 

Total Free Filers, TY 
2018 

Invitation Letters Sent 

Chicago, IL 

60657, 60640, 60618, 
60613, 60625,60660 

4,106 

2,500 


A.2 Recruitment Procedures 

A.2.1 Contact Strategy 

MITRE conducted a multi-stage contact strategy that informed potential participants about the 
study and invited them to participate. First, the IRS sent an introductory letter to the list of 
potential participants identified by MITRE. This letter provided high-level information about the 
study and alerted the taxpayer that they may be contacted by MITRE to participate. The letter 
also provided an IRS toll-free phone number, the internet address for the IRS official website 
(www.IRS.gov), as well as where to find the MITRE study listed on the IRS website so receivers 
of the letter can call or visit to verify the legitimacy of the study. 

The introductory letter from the IRS was mailed to on July 3, 2019. The introductory letter 
followed by a letter from MITRE inviting taxpayers to participate in the study. The MITRE letter 
was mailed on July 8, 2019. 

The MITRE invitation letter provided interested taxpayers a MITRE phone number to call and a 
MITRE-hosted website to visit and input contact information. MITRE took calls, contacted 
taxpayers who submitted information via the website, and screened taxpayers to determine 
eligibility for participation and scheduled interviews. 

A.2.2 Recruitment, Screening, and Scheduling Interviews 

The study team created a workbook that served as a database for all participant contact, 
demographic data collection, and interview scheduling. Interviews were conducted at Harry S. 
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Truman City College in Chicago, IL. This venue has no IRS or MITRE affiliation, is ADA 
accessible, and otherwise allowed ease of respondent access. 

Interested taxpayers called the MITRE phone line or submitted web inquiries from each of the 
four locations as letters arrived. MITRE developed a screener questionnaire that researchers 
administered over the phone to ensure only eligible participants (those whose households had 
received a letter from the pre-selected sample criteria) were able to schedule and participate in 
interviews. The screener also allowed MITRE to collect demographic information that was not 
available from tax return data obtained from CDW such as education level. The MITRE team 
scheduled interviews with participants on the phone, sent confirmation emails when an email 
address was provided, and conducted follow up confirmation calls about 24 business hours 
before the first interviews of each visit were scheduled. 

A.2.3 Response Rates 

Of the 2,500 letters that were mailed out, MITRE received responses from 141 respondents, for a 
total response rate of 5.64 percent. The study relied on taxpayers who expressed interest in 
participating in the study by contacting MITRE after receiving an invitation letter to derive the 
interview sample. MITRE was not able to control or influence who decided to respond to the 
invitation letter upon receiving it. Out of these 141 respondents, the MITRE recruitment team 
screened 42 respondents whom the team was able to reach via phone call, either when the 
respondent called in, or as MITRE attempted to call those who had left voicemails, missed calls, 
or submitted web inquiries. Of the 42 screened, MITRE determined that 36 people were eligible 
to participate in the study due to having received the MITRE letter directly, and thus being in the 
intended sample drawn. If a candidate indicated they filed their individual tax return online at no 
cost but were uncertain if the return was filed through the IRS Free File program, they were 
considered eligible for this study. Table 72 shows that 10 of the 36 eligible candidates screened 
(28 percent) were uncertain if their tax return was filled through the IRS Free File program or a 
commercially available free service. When asked, candidates stated they recalled using a tax 
software program provided by one of the Free File program members, and they were able to file 
their tax return for free. However, these candidates were unsure if the free tax software program 
they used was affiliated with the IRS Free File program. 


Table 72. Free Filing Service Used by Screened Candidates Within Previous Two Years. 



Count of 

Percent of 


Candidates 

Eligible 



Candidates 

Used IRS Free File Program 

26 

72% 

Uncertain IRS Free File or Commercial Program Used 

10 

28% 


Of the 42 respondents screened, five (5) were determined to be ineligible because they did not 
use the IRS Free File program, or commercially available free service and one respondent was 
unable to travel to the interview location within the available timeframes. MITRE scheduled 31 
interviews on a first-come, first-serve basis of respondent availability. This number was both 
within the range of the ideal number of study participants and represented MITRE’s capacity to 
interview. Additionally, MITRE created a waitlist for participant interviews for each site in case 
a minimum necessary number of interviews could not be conducted due to respondents’ failure 
to appear. Two participants failed to appear to their scheduled interview, and one participant 
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cancelled approximately 24 hours in advance of their scheduled interview. MITRE called two 
candidates from the waitlist participants. Twenty-nine total interviews were deemed 
“completed,” meaning that the participants were eligible, and testing took place. This does not 
necessarily mean that a participant completed all tasks required for the session. Table 73 
provides a summary of candidate responses to invitation. 


Table 73. Candidate Response. 



Count 

Percent of Total 



Respondents 

Number of Letters Mailed 

2,500 



Total Number of Respondents 

141 



Number of Candidates Screened 

42 

30% 


Number of Screened Candidates Eligible to Participate 

36 

26% 


Number of Participants Scheduled 

31 

22% 


A.2.4 Demographic Characteristics of Screened Respondents 

Table 74 shows the demographic breakdown of candidates that completed the screening process 
interview and were deemed eligible to participate in the interviews. The data in Table 74 were 
reported by the participant during phone screening, not derived from tax return data in CDW. At 
a glance, the vast majority (92 percent) of respondents filed as single, and the largest age range 
was 31-55 years of age (53 percent) followed by 18-39 (28 percent). Forty-two percent reported 
a household income between $15,001 and $35,000 and the most common educational level was a 
bachelor’s degree (53 percent) followed by having a master’s degree (28 percent). Note again 
that participant recruitment ultimately depended upon the taxpayers themselves contacting 
MITRE after receiving an invitation letter; therefore, MITRE was not able to dictate the 
characteristics of the taxpayers who participated beyond the sample plan described in Section 2.2 
Recruitment. 
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Table 74. Demographic Characteristics of Respondents Screened. 



Count of 

Percent of Eligible 


Respondents 

Respondents 

Number of Screened Respondents Eligible to Participate 

36 


Filing Status of Eligible Screened Respondents 

Single 

33 

92% 

Married Filing Joint 

0 

0% 

Married Filing Separate 

0 

0% 

Head of Household 

3 

8% 

W ido w/W ido wer 

0 

0% 

Unknown 

0 

0% 

Age Range of Eligible Screened Respondents 

18-30 

10 

28% 

31-55 

19 

53% 

56-68 

3 

8% 

69-81 

3 

8% 

82+ 

1 

3% 

Household Income of Eligible Screened Respondents 

Less than $15,000 

7 

19% 

Between $15,001 and $35,000 

15 

42% 

Between $35,001 and $50,000 

8 

22% 

Between $50,001 and $75,000 

5 

14% 

Between $75,001 and $90,000 

1 

3% 

More than $90,000 a year 

0 

0% 

Unknown 

0 

0% 


Highest Level of Education of Eligible Screened 


Respondents 


No schooling completed 

0 

0% 

Some school but did not complete high school 

0 

0% 

Received high school diploma, GED or alternative 

5 

14% 

Associate’s degree 

1 

3% 

Bachelor’s degree 

19 

53% 

Master’s degree 

10 

28% 

Professional degree beyond bachelor’s degree 

0 

0% 

Doctorate degree 

1 

3% 

Unknown 

0 

0% 

Military Pay of Eligible Screened Respondents 

Participant and/or spouse received military pay 

0 

0% 

Zip Codes of Eligible Screened Respondents 

60613 

3 

8% 

60618 

9 

25% 

60625 

4 

11% 

60640 

12 

33% 

60660 

3 

8% 

60657 

5 

14% 
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A.3 Data Collection 

A.3.1 Usability Testing Approach 

To collect data on the Free File system usability, user experience, and satisfaction, MITRE 
conducted 29 one-on-one, in-person testing interviews. These interviews lasted 60-90 minutes, 
and interviews were cut off if the participant had not finished the interview tasks at the 90- 
minute mark. 

In keeping with research best practices [7], the testing approach was designed to meet the needs 
of the research objectives. Traditional usability testing is typically thought to focus on systems or 
interfaces (e.g., websites, software systems), and consists of inviting users to participate in 
representative activities designed to evaluate user errors as well as user satisfaction (how 
pleasant or difficult it is to use the design) [8]; see also [9]. MITRE applied these principles to 
the testing approach. 

To enhance the findings and incorporate more insight into taxpayers’ cognitive processes while 
completing the tasks, the MITRE team also drew on insights from traditional cognitive pretesting 
methods, typically used for evaluating survey and questionnaire design [10], As such, the 
interview protocol included debriefing questions designed to understand taxpayers’ cognitive 
processes as they went through the tasks of finding Free File, selecting a Free File software offer, 
and filing a return. An additional important feature of the usability testing approach was to 
intervene in the taxpayers’ processes as little as possible [11], allowing the user’s experiences 
and challenges to drive the findings as much as possible. 

During each testing task, the interviewer and a notetaker took note of instances where the 
taxpayer expressed frustration, confusion, or hesitation. The notetaker also recorded aspects of 
the taxpayers’ behavior including whether or not they successfully located the IRS.gov Free File 
landing page independently (i.e., without any intervention from the interviewer), whether they 
successfully selected a Free File software offer independently with and without the lookup tool, 
whether errors were committed on the lookup tool and while filing a return, and which Free File 
software offers they selected. During debriefing, the interviewer probed on these behaviors to 
understand what was going through the taxpayer’s mind and how the taxpayer interpreted 
elements of the Free File system. 

The interview protocol followed this format: 

1 . Introduction. The interviewer explained that the purpose of the interview is to evaluate 
the design of the Free File system, not evaluate the taxpayers’ skills or abilities. The 
interviewer explained that MITRE did not design any of the software systems tested and 
stressed their own role as an observer, not a helper. The interviewer also explained that 
MITRE is a not-for-profit organization and has no commercial interest in the Free File 
program. The taxpayer signed an informed consent form. 

2. Usability Task 1: Locating Free File. The taxpayer was not presented with a fictitious 
scenario for this part of the interview. The taxpayer was asked to imagine that they 
learned that the IRS offered a program to file federal returns for free, and how they would 
go about locating that service. The interviewer and notetaker observed the taxpayer 
complete the task. The interviewer allowed the participant to continue independently up 
until the point of selecting a Free File software offer for their own eligibility situation, if 
and only if the taxpayer located the Free File landing page independently. If the taxpayer 
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indicated that they would end on a commercial website or were not able to find Free File, 
the interviewer intervened and directed them to IRS.gov and asked the participant to 
search from there. 

3. Single Ease Question (SEQ 1) and Debriefing Questions on Locating Free File task. 

The interviewer asked the participant to fill out the SEQ first question to rate the ease of 
the task. The interviewer then asked a series of questions designed to better understand 
the taxpayer’s actions and experiences, such as, “Can you tell me more/walk me through 
the process you use to find the location where you file for free?” 

4. Usability Task 2a: Selecting a Free File Software Offer, Scenario 1. The interviewer 
directed the taxpayer to the Free File landing page and gave the taxpayer a fictitious 
scenario (two scenarios were used; their order was alternated, see section A.3.2). The 
taxpayer was tasked with selecting an eligible software offer for the fictitious person. 

5. Usability Task 2b: Selecting a Free File Software Offer with the Lookup Tool, 
Scenario 1 (if needed). If the taxpayer did not independently select to use the lookup tool 
in test 2a, the interviewer asked the participant to repeat scenario 1 and use the lookup 
tool. 

6. Usability Task 2c: Selecting a Free File Software Offer with the Lookup Tool, 
Scenario 2. The interviewer administered a second lookup tool scenario (designed to 
return a different set of results if completed correctly). 

7. SEQ 2 and Debriefing Questions on Usability Task 2. The interviewer asked the 
participant to fill out the SEQ second question to rate the ease of the task. The 
interviewer then asked a series of questions designed to better understand the taxpayer’s 
actions and experiences, such as how they decided whether or not to use the lookup tool. 

8. Usability Task 3: Filing a return. The interviewer instructed the taxpayer to select one 
of three randomly chosen Free File software offers to test. The interviewer gave the 
taxpayer a new fictitious scenario, the filing scenario. The taxpayer created an account 
and filled out a return using the filing scenario. 

9. SEQ 2 and Debriefing Questions on Usability Task 3. The interviewer asked the 
participant to fill out the SEQ third question to rate the ease of the task. The interviewer 
then asked a series of questions designed to better understand the taxpayer’s actions and 
experiences, such as “Overall, how easy or difficult was your experience using the Free 
File program?” 

10. Usability Metric for User Experience (UMUX) -Lite Questionnaire. All participants 
filled out a satisfaction questionnaire to measure user satisfaction and other aspects of 
user experience. 

11. Overall Debriefing Questions. Interviewers asked all participants (regardless of how 
many vignettes were given) a series of questions about their past experiences with the 
Free File program, how they heard about Free File, how their prior experience compared 
to the testing situation, and for suggestions for how the IRS could make the Free File 
experience easier for taxpayers. 
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12. Incentive and Closeout. The interviewer gave the taxpayer a $100 prepaid VISA gift 
card and thanked them for their time. The taxpayer signed a voucher confirming they 
received the gift card. 

A.3.2 Test Scenarios 

In preparation for the usability interviews, the MITRE team developed three fictitious scenarios. 
Two scenarios were used to support tasks where taxpayers were asked to select a Free File 
software offer for which a fictitious person was eligible (selection scenarios 1 and 2). The third 
scenario was used to support the task where taxpayers were asked to complete a tax return using 
a Free File software offer chosen by MITRE (filing scenario). To design the selection scenarios, 
MITRE examined eligibility criteria for different Free File software offers and developed 
taxpayer characteristics that would return different sets of possible Free File software offers. To 
design the filing scenario, the team had two goals: 1) to introduce enough complexity into the 
scenario so that taxpayers would have to interact with multiple components of the tax preparation 
software, including filing status, dependents, and adjustments to income, while still keeping the 
scenario simple enough to complete in approximately 30 minutes, and 2) to ensure that scenarios 
represented feasible free filer filing characteristics. Using CDW data, the team identified which 
fields on previous were commonly populated by individuals filing through Free File. In 
particular, MITRE’s review of the free filer data available in CDW shows (see Appendix D for 
detailed descriptive statistics): 

• 71 percent of taxpayers who filed through the IRS Free File program in TY 2018 reported 
a filing status of single; the second most common filing status was head of household (17 
percent) 

• Mean AGI for free filers in TY 2018 was $23,247 

• Thirty-five percent of taxpayers using the IRS Free File program in TY 2018 claimed any 
non-zero amount for Earned Income Tax Credit (EITC), with a mean EITC distribution 
value of $566 

• Exactly 1.6 percent of taxpayers using the IRS Free File program in TY 2018 itemized 
deductions 

All scenarios used a fictious name, address, and other taxpayer information. Any resemblance to 
actual persons, living or dead, is purely coincidental. A summary of each scenario is provided 
below. See Appendix A for the complete set of scenarios used during the usability study. 

• Selection Scenario 1. Participants used this scenario to identify and select a Free File 
software offer using the IRS Free File program web page (which may have included using 
the software lookup tool). The scenario contains details about a single taxpayer, age 52, 
who resides in Massachusetts, has one dependent, and 2018 AGI of $49,000. This scenario 
was designed to return eligibility results that do not include the large company software 
providers, which together represent approximately 70 percent of Free File returns filed in 
TY2018F 


16 Based on data from CDW. 
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• Selection Scenario 2. Participants also used this scenario to identify and select a Free File 
software offer using the IRS Free File program web page (which may have included using 
the software lookup tool). The scenario contains details about a single taxpayer, age 43, 
who resides in Texas, has no dependents, and 2018 AGI of $33,000. This scenario was 
designed to return eligibility results that do not include options to file a state return for 
free. 

• Filing Scenario. Participants used this scenario to complete an online federal tax return 
using a Free File software offering selected at random by MITRE. The scenario contains 
[12] and student loan interest paid as reported on Form 1098-E. This taxpayer is also 
eligible for the EITC. The scenario included additional information on the payment of 
household expenses to introduce some complexity as to whether or not the taxpayer should 
file as single or head of household (the two most co mm on filing statuses for free filers). 
This component of the scenario pushed taxpayers to interact more fully with the filing 
status feature of the tax preparation systems. 

Selection scenarios 1 and 2 were required to complete the tasks related to the Free File software 
lookup tool. To control for order effect, MITRE alternated the order that participants received 
selection scenarios 1 and 2. All participants completed the same filing scenario. 

A.3.3 Satisfaction and Overall Experience Questionnaires 

After completing each task, participants responded to the SEQ regarding that task. After 
completing all tasks, users answered the UMUX about their entire Free File experience during 
the session. These questionnaires gathered self-reported satisfaction and usability data about 
participant experiences using the IRS Free File program. Note that when usability and ease of 
use are discussed in this section, they are actually indicative of perceived usability and perceived 
ease of use; they do not comprehensively take into account other aspects of usability like 
effectiveness and efficiency. 

The SEQ was created by Sauro and Dumas [13]. The SEQ is a single-item questionnaire that 
measures user perception of ease related to specific tasks. It is asked directly after the relevant 
task and is administered on a seven-point response scale, where one is “very difficult” and seven 
is “very easy.” SEQ scores are interpreted against an industry-accepted scale of below-average, 
average, and above-average SEQ response ranges. 

• Item: “Overall, how easy or difficult was the task to complete?” 

The UMUX was created in 2010 by Finstad [6]. It is a four-item questionnaire that looks at an 
entire experience; it is not task-specific. Its four questions for measuring usability are targeted at 
the ISO 9241 definition of usability (effectiveness, efficiency, and satisfaction). There are two 
positive and two negative items, all administered on a seven-point response scale, where one is 
“strongly disagree” and seven is “strongly agree” with the item in question. “Free File software 
system” was substituted for “this system.” UMUX responses are converted to scores and 
compared to an industry-accepted acceptability and “grading” comparison system (e.g., a score 
in the C range is marginally acceptable). UMUX metrics are often used to track usability of a 
system and set future usability goals. Here, the UMUX is used to assess usability of the entire 
experience rather than of individual tasks. 

• Items: 
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o “[This system’s] capabilities meet my requirements.” 
o “Using [this system] is a frustrating experience.” 
o “[This system] is easy to use.” 

o “I have to spend too much time correcting things with [this system].” 

A.3.4 Taxpayer Perceptions 

At the end of each interview, the interviewer asked additional debriefing questions to get the 
participant to reflect on their experience with Free File. These questions were intentionally asked 
at the end of the exercise, as asking for this type of information upfront could have primed the 
participant to think about these experiences prior to completing the Free File tasks. Debriefing 
questions included the following topics: 

• Taxpayers’ prior experience using Free File 

• How taxpayers heard about Free File and what vendor(s) they previously used 

• How the IRS could make the Free File program easier to use 

• Type(s) of help the participant would look for (call the IRS, website, etc.) 

During project planning, the sponsor indicated interest in understanding whether or not the 
interests of military taxpayers are represented by the Free File system. MITRE had already 
decided to conduct interviews in Chicago zip codes at this point and did not expect military 
member participation based on the invitation letter sample, so MITRE included a debriefing 
question after the Free File software offer selection task to ask, “If you were a member of the 
military, what would you think about this site? What steps would you take? [if needed] Would 
you have done anything differently?” 

Note that the only time taxpayers were asked to reflect on their prior experiences with Free File 
was in the overall debriefing interview. However, some taxpayers brought up this prior 
experience in the context of the usability testing tasks. Their relevant comments and 
recommendations are reported throughout this report. 

A.3.5 Materials for Data Collection 

The following materials were used for the usability testing sessions and are included in Appendix 
C: 

1. Letter from IRS 

2. Letter from MITRE 

3. Screening protocol 

4. Interview protocol 

5. Laptop with multiple web browsers loaded (Internet Explorer, Google Chrome, and 
Firefox) and Morae recording software installed 

6. Informed consent form 

7. SEQ questionnaire with three questions 

8. UMUX-Lite questionnaire 
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9. Test scenarios 

10. Mocked-up tax documents for test scenarios (not attached; they are described in the test 
scenario document) 

11. Receipt form for honorarium 

A.3.6 Note-Taking Procedures 

Each interview involved two MITRE researchers: an interviewer and a notetaker. The 
interviewer was tasked with administering the interview, including obtaining informed consent, 
explaining the usability activities, asking debriefing questions, and administering the 
questionnaires. The interviewer was responsible for using his/her own judgment to effectively 
follow-up on participant points of confusion. The notetaker was responsible for capturing the 
participant’s behaviors and statements in a predefined notetaking template throughout the 
interview. Notetakers were also able to ask the participant questions in order to clarify aspects of 
the interview, if not covered by the designated interviewer. 

A.4 Analytical Approach 
A.4.1 Usability Data 

For the purposes of this study, usability data are the observations made by the MITRE 
researchers while participants completed the test tasks (locating Free File, selecting a Free File 
software offer, and filing a return). These data are focused on observations of what the 
participant did while completing the tasks, aspects of completing the tasks that caused hesitation, 
confusion, or frustration, minor and major errors, participant comments that came up while they 
were in the process of completing the tasks, and occasions where the participant abandoned the 
task or indicated that they would abandon the task in a real setting. 

To analyze these data, two MITRE team members read through each usability session summary, 
broken out by test task. They identified problems, confusion points, and errors, and extracted 
these issues into a running list of usability issues. As new issues were observed in the usability 
summaries, they were added to the list; as the researchers observed new instances of existing 
issues, they aggregated them onto the list. The researchers added references to each unique 
participant that experienced a given issue and used these participant ID lists to generate 
frequency counts of how often taxpayers experienced the issue. 

The two researchers responsible for this part of the data analysis worked independently, with 
regular check-ins and regularly reviewed each other’s analyses. A third researcher reviewed all 
usability issues for quality control purposes. 

The team gave each issue a severity rating, described below. These severity ratings allow a quick 
reference to gauge how serious an issue was—was it a showstopper for some taxpayers, or 
merely a point of confusion? Did errors result for some taxpayers who encountered this problem? 
Note that severity levels do not necessarily tie to the frequency of the issues among the 
participants and are treated as distinct concepts [2], 17 


17 Severity is not tied to frequency because regardless of how many people experience a high severity issue, it is 
problematic if anything about the system design causes taxpayers to be unable to complete the task. 
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1 . High severity issues. Issues that caused users to commit major errors or stopped the 
taxpayer from completing the task at hand. For example, entering an incorrect amount 
that causes cascading errors to throughout the tax return and to the final refund amount. 

2. Medium severity issues. Issues that caused major frustration that the participant was 
able to eventually resolve on his/her own, or issues that caused minor errors. For 
example, not ticking a required checkbox but otherwise recording other amounts 
correctly. 

3. Low severity issues. Issues that caused some frustration or caused the user to express 
dissatisfaction with the experience of completing the form, but the responses were correct 
in the end. 

A.4.2 Taxpayer Pathways Analysis 

To better understand how taxpayers navigated the task of finding the Free File page online and 
select a Free File software offer, MITRE used an approach called user flow analysis. User flow is 
a series of steps that a user performs to complete a task during an interaction with the system of 
interest. It is the path a user follows through the system. 

User flow analysis is used to give insight into which paths more commonly lead to a desired 
outcome, such as learning X information, signing up for an e-list, completing an account 
creation, or completing an online purchase. It helps researchers see where users arrived from, 
what actions they took, and where they exited the system. It can also show which parts of the 
process commonly cause confusion for users, and where users get stuck, waste energy, or 
abandon the task. It ultimately reveals opportunities for inefficiency and error in the current user 
interface, like unclear menus or unexpected placement of pages within the information 
architecture. Once these negative opportunities are known, they can be changed to enable users 
to reach the desired outcome more often. 

Also, armed with the knowledge of which pages receive the highest traffic (the most “eyeballs”) 
in a system, one can move the most important information or elements into those areas to ensure 
they are seen more often. Many companies employ user flow as a form of analytics, and Google 
Analytics offers user flow information as well. 

MITRE collected participant pathways by observing the actions they took throughout their 
session and the comments they made. The team recorded unique pathways taken by taxpayers 
during two flows: locating the Free File landing page; selecting a Free File software offer for 
themselves (among taxpayers who successfully located the Free File landing page 
independently). The team tabulated the frequency of different pathways. 

Overall, the team did the following in this analysis: 

• Compared the routes participants took to complete the tasks 

• Identified areas of common confusion, inefficiency, and error 

• Demonstrated the taxpayer decision-making path 

• Gained understanding on how efficiently the system is driving users to the goal action(s) 
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A.4.3 Taxpayer Behavior Metrics and Search Terms 

In addition to recording the user flows that taxpayers followed in the usability tasks, MITRE 
captured and reported on the following metrics: 

• Number of participants who opted to use the software lookup tool 

• Errors committed using the software lookup tool 

• Vendors selected by taxpayers during the testing session 

• Search keywords taxpayers used to find the Free File landing page 

MITRE used the last data point, search keywords used to find the Free File landing page, in 
conjunction with findings from the Google Web analytics analysis MITRE conducted as a 
separate piece of the Free File Program Assessment Final Report to evaluate whether and how 
MITRE’s study materials (recruitment letters, interview protocol, etc.) may have influenced how 
participants searched for Free File in the usability task. This analysis was done ad hoc, the study 
was not designed to definitively test whether the usability participants’ search behaviors are 
representative of the general taxpayer population, or to measure the extent to which interview 
procedures affect participant behavior. As such, MITRE’s conclusions on this topic should be 
interpreted with caution. 

A.4.4 Qualitative Data 

The analysis team used a limited content analysis approach to analyze the qualitative data. Two 
analysts examined a specific subset of the data in which taxpayers were asked to describe their 
experience with Free File prior to the interview, a comparison between their prior experience and 
their Free File experience during the testing session, and a debriefing probe during the course of 
the interview about how they would have selected a Free File software offer if they were in the 
military. 

Coding is the fundamental process of converting open text to usable information by pulling out 
repeated phrases, terms, and ideas from a corpus of data and identifying commonalities, 
intersections, and other interactions among them. Coding can also be referred to as tagging or 
marking depending on the methodology and approach of a piece of source material. 

The limited size and scope of the dataset prevented the analysts from delving into any set of 
questions to draw out underlying motivations, justifications, or meaning beyond the stated 
reasons provided by participants. Instead, the results of the content analysis were used “as is” 
and reflect a simple count of the types of responses provided by participants. 

Qualitative analysis of any sort has several inherent limitations. First, there is no underlying 
distribution of data that can be incorporated into a statistical analysis. Each unit of data can be 
associated with multiple codes, and each code can be associated with multiple units of data. In 
addition, a lack of any codes in a unit of data simply means that a respondent did not mention 
that topic, it does not mean that the topic is not relevant to the respondent. 
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A.4.5 Questionnaire Analysis 

To analyze responses, SEQ and UMUX data were grouped into subsets for like-comparison 
purposes, ex. grouping within certain age ranges. Due to the small sample size and tight project 
timeline, data was not filtered, and statistical significance was not tested for. 

In demographic-focused charts, a dotted line is used to indicate groups that contained only one 
participant. These one-participant groups are excluded from discussions on demographic-based 
comparisons due to their small sample sizes. 

A.4.5.1 Single Ease Question 

MITRE examined mean SEQ responses, which are presented overlaid with typical SEQ response 
bands [14] [5] [15] (below-average, average, and above-average). These bands are taken from 
SEQ data from many industry and academic usability studies on a variety of interfaces. 

SEQ Percentile Rank: 10 tn 20 tn 

TaskCompletionRate: 47% 51% 

TaskTime: 3.4 min 2.8 min 


SEQ Raw Score: 1 1.5 2 2.5 3 3.5 4 

Very Difficult 

Figure 41. SEQ Score Bands. 

SEQ score bands [14] 

A.4.5.2 Usability Metric for User Experience 

UMUX responses were converted into scores using the following process [6]: 

• Odd items are scored as (score-1), even items are scored as (7-score), to remove the 
positive/negative keying of items and enable a minimum score of zero. After recoding, 
each item has a range of 0-6, and the entire scale has a preliminary maximum of 24. 

• A participant’s UMUX score is the sum of the 4 items divided by 24, then multiplied by 
100. This conversion gives a 0-100 range and allows comparison to System Usability 
Scale (SUS) metrics, an older and longer usability metric. 

In brief, the conversion equation is: 

UMUX = UUMUXiteml - 1) + (UMUXitemS - 1) + (7 - UMUXiteml) + (7 - UMUXitem4))x — 

24 

The average score across participants is then taken as the UMUX metrics of the system of 
interest. Responses to individual items averaged across participants are examined for more 
detailed insights. To interpret scores, the processed UMUX scores are treated like SUS scores. 
Previous work [16] developed percentile, grading, adjective [17], and acceptability [18] 
descriptors for understanding SUS scores, which we use. In charts, mean UMUX processed 
scores are presented overlaid with adjectival and grade UMUX response bands to guide 
interpretation. 


30 tn 40 tn 50 tn 60 tn 70 tn 80 tn 90 tn 99 tn 

58% 68% 71% 80% 86% 87% 90% 91% 

2.8 min 2.2 min 2.0 2.1 2.0 1.9 1.6 1.1 min 


4.5 
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NPS: 


Detractor 


Passive Promoter 


_ . , Not Acceptable Marginal Acceptable 

Acceptable: r mu * m mm 

... .. Worst Imaginable Poor OK Good Excellent Best Imaginable 

Adjective: |||||1I|||||||||||||1III||||||1 I II INI III III HI MH HI II 


Grade: F D C B A 


sus Score: 0 10 20 30 40 50 60 70 80 90 100 


Figure 42. Grades, Adjectives, Acceptability and Net Promoter Score. 

Grades, adjectives, acceptability, and Net Promoter Score (NPS) categories associated with raw 
SUS scores (equivalent to processed UMUXscores) [16] [11]. 

A general guideline is provided in Table 75 below [19]: 


Table 75. General Guidelines for SUS Scoring. 


sus 

Score 

Adjective 

Rating 

Acceptability 

92 

Best 

imaginable 

Acceptable 

85 

Excellent 

Acceptable 

72 

Good 

Acceptable 

52 

OK/Fair 

Marginal 

38 

Poor 

Not 

Acceptable 

25 

Worst 

imaginable 

Not 

Acceptable 

SUS 

Score 

Grade 

Acceptability 

>81 

A 

Acceptable 

68-81 

B 

Acceptable 

68 

C 

Marginal 

51-67 

D 

Marginal 

<51 

F 

Not 

Acceptable 
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Appendix C Recruitment and Interview Materials 


Department of the Treasury Date: July 2.2019 

Internal Revenue Service 

c/o The MITRE Corporation 
7525 Colshire Drive 
McLean, VA 22102 


Dear [ , 

The IRS needs your help with an important study to better understand how taxpayers interact with our Free File 
online software program. We’ve contracted the MITRE Corporation, who 11 ms the IRS’s Federally Funded 
Research and Development Center, to meet with taxpayers to see how they use IRS Free File. 

We need volunteers in your city to participate in an in-person interview. Participation will take 60-90 minutes 
and MITRE will give you a $100 gift card in appreciation of your tune. 

In the next few days, you’ll receive a letter from MITRE inviting you to participate in the study. They won’t ask 
you for your personal tax information. 

If you want to verify this study is legitimate and not a scam, you can visit www.irs.gov css . Scroll down to the 
bottom of the IRS.gov Customer Satisfaction Surveys page and find this IRS Wage and Investment (W&I) Free 
File study. You may also call us at 800-829-1040. 

Please consider taking part in this important study. Thank you in advance for your time and feedback. 

Sincerely, 

CiwtKueter 

Curt Rueter 

Deputy Director, Submission Processing 


The Paperwork Reduction Act provides that the IRS may not conduct or sponsor, and you are not required to respond to a collection of 
information unless it displays a vahd OMB control number. The OMB Control Number for this study is 1545-2274. The time 
estimated for your participation is 90 minutes. If you have any comments on the time estimates associated with this study or 
suggestions on making this process simpler, please write to the Internal Revenue Service. Tax Products Coordinating Committee. 
SE:W:CAR:MP:T:T:SP. - Room 6129 1111 Constihition Ave.. NW. Washington. DC 20224. 

Letter 6231 (6-2019) 

Catalog Number 72727M 


Figure 43. IRS Invitation Letter. 
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MITRE 

7525 Colsliire Drive 

McLean, VA 22102 

My 8,2019 




Dear L 


In the last few days, you should’ve received a letter from the Internal Revenue Seivice (IRS) asking you to help 
them with MITRE’s important study on the IRS Free File online software program. The MITRE Corporation is a 
not-for-profit company that conducts research and development for the federal government. 

We invite you to participate in this study as a taxpayer who is eligible to use IRS Free File. We need 
volunteers in your city to participate in an in-person interview. Participants will receive a S100 prepaid VISA 

gift card. 

How to participate 


• Please call (571) 310-1902 or visit www.mitre.org/freefile. 

• We’ll ask some questions to determine eligibility for in-person interview. We won’t ask for your Social Security 
number or individual taxpayer identification number. 

• We’ll schedule a time to meet with you for a 60 to 90-minute in-person interview, if you’re eligible to 

participate. 


Why we’re conducting this study 


• We need your feedback to learn whether people can easily use IRS Free File online software. 

• We’ll use your feedback to recommend how the IRS can improve the user experience of the IRS Free File online 

program. 


If you want to verify this study is legitimate and not a scam, vou can visit www.irs. 20 v/css. Scroll down to the 
bottom of the IRS.gov Customer Satisfaction Surveys page and find this Wage and Investment (W&I) Free File 
study. You may also call the IRS at (800) 829-1040. We’re conducting this study under Office of Management and 

Budget approval number 1545-2274. 


Thank you for your consideration. 

Sincerely, 


LuoLcv Lykkes 

MITRE Study Team Members: 

Lucia Lykke, Ph.D. 

Jamel Morris, MITRE Project Manager 

Lead Researcher 

Mikhael King 

The Paperwork Reduction Act provides that the IRS may not conduct or sponsor, and you are not required to respond to. a collection of information 
unless it displays a valid OMB control number. The OMB Control Number for this study is 1545-2274. The time estimated for your participation 
is 90 minutes. If you have any comments on the time estimates associated with this study or suggestions on making this process simpler, please write 
to the Internal Revenue Seivice. Tax Products Coordinating Committee, SE:W:CAR:MP:T:T:SP. - Room 6129 1111 Constitution Ave.. NW. 

Washington. DC 20224. 

Letter 6231-A (6-2019) 

Catalog number 72738X 


Figure 44. MITRE Invitation Letter. 
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MITRE 


ABOUT CENTERS CAPABILITIES RESEARCH CAREERS PUBLICATIONS NEWS 


a 


Online Screening Form for Jaxpayers 

Welcome to the MITRE Free File Program Study site. Thank you for visiting and answering a few questions to help us determine if you are 
eligible to participate. 

MITRE is conducting a study to understands taxpayers’ experience with the IRS Free File online software program. MITRE is looking for 
volunteers in your area to participate in a 60- to 90-minute, in-person research interview. Participants will receive a $100 pre-paid VISA gift card 
as a token of our appreciation. 

As a taxpayer who may use IRS Free File, we believe you might be eligible to participate. The questions below will help us determine eligibility. 
* Asterisks mark required fields 

First Name * Last Name * 


If you qualify, we will call you to schedule your interview and to ask a few more eligibility questions. 

What Is the best phone number to reach you? * What is your email address? 

What Is the best time of day for us to call you? Morning Afternoon Evening 
In what zip code do you currently live? * 

Did you receive a letter from MITRE asking you to participate in this study? * Yes No Not sure 

Thank you for your responses. We will be in touch soon if you are eligible to participate in the independent research study. 
For any questions about this study, please contact the MITRE team at (571) 310-1902. 

To verify the this study with the IRS, visit www.irs.gov/css and look for "Free File Study” near the bottom of the page. 

A 

I'm not a robot 

r*CAPTCHA 


Submit 


Figure 45. Screenshot of MITRE Recruitment Website. 

Note: At the conclusion of recruiting, the online entry from was removed from the web page and 
the following notice posted, “Note: Registration is full for this study and we are no longer taking 
additional participants. Thank you for your interest.” 
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MITRE 

Participant Consent Form 

ADULT CONSENT TO PARTICIPATE IN A RESEARCH STUDY 

PRINCIPAL INVESTIGATOR(S): Lucia Lykke, Ph.D.; Jamel Morris, Project Manager 
STUDY TITLE: IRS Free File Program Study 

INTRODUCTION 

We invite you to take part in a research study conducted by The MITRE Corporation (MITRE). 

We want you to know that taking part in MITRE research is entirely voluntary. 

You may choose not to take part, or you may withdraw from the study at any time. 

Before you decide to take part, please take as much time as you need to ask any questions and discuss this study 
with anyone at MITRE, or with family, friends, or any of your advisers. 

THE RESEARCH STUDY 

1. Research Protocol 

The purpose of this research is to learn how taxpayers experience the IRS Free File online software. You will be 
given fictitious taxpayer information and asked to complete an online tax return using this fictitious information. 
You will then be asked some questions about your experience. You will not be asked to provide your personal 
financial information. Participation in this study is expected to last 60-90 minutes. 

2. Risks/ Discomforts 

There are no expected risks or discomforts to the participant involved in this study. 

3. General or Participant Benefits 

You may receive no benefit from taking part. The research may give us knowledge that may benefit the work of 
MITRE and its Sponsors, including the Internal Revenue Service (IRS). 

OTHER PERTINENT INFORMATION 

1. Confidentiality. The identities of participants will not be made known to MITRE management or to IRS 
employees. When results of MITRE research are reported to the IRS or other government agencies, in 
professional journals, or at scientific meetings, the people who take part are not named and identified. Any data 
used is constructed so as to preclude identifying participants. 

2. Recording. This interview may be recorded by the researcher with your consent. The recording will not identify 
you by name, and will be destroyed upon completion of this research study. 

3. Remuneration. As a token of appreciation, you will receive a $ 100 prepaid gift card upon completion of this 
research interview. 

4. Problems or Questions. If you have any problems or questions about this study, or about your rights as a 
research participant, or about any research-related injury, contact the Principal Investigator, Lucia Lykke, at 
(703) 983-7720. Other researchers you may call include: Jamel Morris, at (703) 983-6511. 

5. Consent Document. Please keep a copy of this document in case you want to read it again. 

Participant's Consent 

I have read the explanation about this research study and have been given the opportunity to discuss it and to ask 
questions. I hereby consent to take part in this study. 


Signature of Participant Date 


Signature of Principal Investigator/Researcher Date 


Figure 46. Participant Consent Form. 
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Free File Taxpayer Testing - Selection Scenario 1 


Name: Isaac Washington 
DOB: June 30 ; 1967 (52 years old) 

Employment: Sales Manager 

Marital Status: Single 

State of residence in 2018: Massachusetts 
State return to file: Massachusetts 


You are single with one dependent. You are not in the military and do not have a spouse in 
the military. 

Your adjusted gross income (AGI) in 2018 was $49,000. 


Please find a Free File offering to file your federal return for free using the information above. 


Figure 47. Selection Scenario 1. 

Note: All scenarios used fictious name, address and taxpayer information for the purpose of this 
study’s usability testing. Any resemblance to actual persons, living or dead, is purely 
coincidental. 
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Free File Taxpayer Testing - Selection Scenario 2 


Name: Rita Meyer 

DOB: April 2, 1976 (43 years old) 

Employment: Outreach Coordinator 

Marital Status: Single 

State of residence in 2018: Texas 
State return to file: None 


You are single with no dependents. You are not in the military and do not have a spouse in 
the military. 

Your adjusted gross income (AGI) in 2018 was $33,000. 


Please find a Free File offering to file your federal return for free using the information above. 


Figure 48. Selection Scenario 2. 

Note: All scenarios used fictious name, address and taxpayer information for the purpose of this 
study’s usability testing. Any resemblance to actual persons, living or dead, is purely 
coincidental. 
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Free File Taxpayer Testing - Filing Scenario 

NOTE: DO NOT SUBMIT THIS FICTITIOUS RETURN AT THE END OF THE SCENARIO. 

Name: Christine Coleman 
DOB: July 30, 1990 (28 years old) 

SS#: 011-01-8948 

Employment: Sales Representative 

Marital Status: Single 

Address: 1820 Pine Rd., Houston, TX 77008 

Bank information: Routing #xxxx00059, Checking Account xxxx410201 

You are single with one dependent, your daughter Stephanie. No one else can claim you as a 
dependent. You are not in the military or a veteran. 

When it is time to file your 2018 income tax return, you file your return for yourself. In 2018, 
you and your daughter had full-year health care coverage through your employer. You lived 
in the state of Texas for the entire year and are a U.S. Citizen. 

On your return, you claim the standard deduction for a single taxpayer. 

Your daughter Stephanie is your dependent. She lived in the house with you all year and you 
supported her during the tax year. You and Stephanie's other parent have an agreement that 
only you list her as a dependent. No one else lives with you and Stephanie. Your parents 
helped you out by paying for more than half of the cost of upkeep on your home in 2018. 


Name 

SS# 

DOB 

Relationship 

Stephanie Smith 

459-47-2658 

04/20/2016 

Daughter 


In 2018, you were employed and received the following income: 

• Your employment income from American Standard Sales (Form W-2). 

You have no other forms of income for the tax year. 

You also paid student loan interest in 2018. 

• During 2018, you paid $800 in interest on your qualified student loan to EdFinancial Services (Form 
1098-E) 

• You determined you can deduct the full $800 of the interest paid on your tax return. 

Please complete your online tax return with the information above and the supplemental 
documentation provided to you. 


Figure 49. Filing Scenario. 

Note: All scenarios used fictious name, address and taxpayer information for the purpose of this 
study’s usability testing. Any resemblance to actual persons, living or dead, is purely 
coincidental. 
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Participant ID: 


Usability Single Question Evaluation 

Task 1. Navigating the Internet to file your federal return for free 

Overall, how difficult or easy was this task to complete? 

1 2 3 4 5 

Very 
difficult 


Task 2. Using the Software Provider Lookup Tool to pick a software vendor 

Overall, how difficult or easy was this task to complete? 

1 2 3 4 5 6 

Very 
difficult 


Task 3. Completing a return using a software provider 

Overall, how difficult or easy was this task to complete? 

1 2 3 4 5 

Very 
difficult 


Very easy 


Very easy 


Very easy 


Figure 50. Single Question Evaluation. 
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Free File Program User Experience Questionnaire 

1. This Free File system’s capabilities meet my requirements. 

1 2 3 4 5 

6 

7 

Strongly 


Strongly 

Disagree 


Agree 

2. Using the Free File software system is a frustrating experience. 

1 2 3 4 5 

6 

7 

Strongly 


Strongly 

Disagree 


Agree 

3. The Free File software system is easy to use. 

1 2 3 4 5 

6 

7 

Strongly 


Strongly 

Disagree 


Agree 

4.1 have to spend too much time correcting things with this Free File software system. 

1 2 3 4 5 

6 

7 

Strongly 


Strongly 

Disagree 


Agree 


Figure 51. User Experience Questionnaire. 
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MITRE 

Date: 

I,_ (participant’s full name), confirm that I 

received a $100 prepaid VISA gift card as an honorarium for my participation in the MITRE 
Free File research study. 

VISA card number (last 4 digits on back of card) 

Participant’s Signature:_ 

MITRE Researcher’s Signature: 


The MITRE Corporation 

Figure 52. Honorarium Receipt. 
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IRS Free File Program Usability Testing 

Screener and Phone Guide 

Intro 

Thank you for your call, this is [NAME] with the MITRE Corporation. 

i. Did you receive a letter about the Free File Program study that we are doing for the IRS? 

( ) Yes [Record in Spreadsheet] 

( ) No [If no, determine how participant learned of study/why calling] 

ii. How did you learn about this study? 

[If answer to question i is no]: 

We appreciate your interest in our independent study. We are trying to understand a very specific group 
of taxpayers and sent letters to members of that group. If you did not receive a letter, you are not 
eligible to participate. Thank you for calling. Have a nice [day/evening]. 

[If answer to Question i is yes]: 

We are conducting an independent research study for the IRS to gather feedback from taxpayers like 
you, and we are traveling to your area to conduct this research. We are looking for eligible volunteers to 
participate in this study. Eligible volunteers will participate in a 60- to 90-minute in-person interview at a 
public location near you. If eligible, you will receive a $100 Visa gift card as a token of appreciation for 
your participation. We will not ask you for your Tax ID number or Social Security Number during this call, 
or during the in-person interview. 

Do you have about five minutes to answer some questions, so we can verify your eligibility to participate 
in the study? If eligible, we will go ahead and schedule an in-person interview. 

Screener 

1. What is your full name? 

( ) [Record in Spreadsheet] 

2. What is the best contact phone number for you? 

( ) [Record in Spreadsheet] 

3. Do you have an e-mail address that you would like to use for any future correspondence about the 
study? 

( ) [Record in Spreadsheet] 

4. What is the zip code of your primary residence? 

[Record zip code on spreadsheet] 

( ) Within set zip code limits of travel location 

( ) Outside of set zip code/city limits - Terminate: I'm sorry, we are only conducting interviews 
within a certain set of zip codes, so you don't qualify for this study. Thank you for your time. Have a 
good [doy/evening]. 


C-ll 

PRE-DECISIONAL - FOR INTERNAL IRS W&l USE ONLY - NOT FOR PUBLIC RELEASE 



PRE-DECISIONAL - FOR INTERNAL IRS W&l USE ONLY - NOT FOR PUBLIC RELEASE 


5. We will be conducting sessions at your location between July 22 and July 26. We are looking for 
participants to meet us at Truman College for a 60 to 90 minute interview. Are you available to meet us 
at Truman College during this time frame? 

( ) Yes - Continue 

( ) No - Terminate: Thank you for your time. Have a good [day/evening]. 

6. Within the last two tax years (2017, 2018), have you filed your federal tax return through the IRS Free 
File Program? 

( ) Yes - Continue to next question 

( ) No - Terminate: I'm sorry, we are only looking for people who have previously filed through the 
IRS Free File Program. Thank you for your time. Have a good [day/evening]. 

( ) Not sure - Ask caller if they filed for free or if they paid any fees to file. 

If they filed for free but not sure if it was through Free File, continue to next question. 

Record "Not Sure" in Spreadsheet Column H and record what they say in Column I. 

If they paid any fees, terminate. We are looking for participants who filed through the IRS 
Free File Program. Thank you for you time. Have a good [day/evening]. Record "Not Sure" in 
Spreadsheet Column H and record what they say in Column I. 


7. Would you say that you are the adult in your household who is most familiar with the preparation 
and filing of your federal income tax returns? [ If nol Would you say you are equally familiar as the adult 
most familiar with the preparation and filing of your federal income tax returns? 

( ) Yes - Continue 
( ) Equally familiar-Continue 

( ) No - Terminate: I'm sorry, we are only conducting interviews with people who are familiar with 
their tax form. Thank you for your time. Have a good [day/evening]. 

8. What was your filing status for tax year 2018? 

( ) Single 

( ) Married Filing Joint 
( ) Married Filing Separate 
( ) Head of Household 
( ) Widow/Widower 

9. What is your age? 

( ) under 18- Terminate: Thank you for your time. Have a good [day/evening]. 

( ) 18-30 
( ) 31-55 
( ) 56-68 
( ) 69-81 
( ) 32+ 

10. Is your current household's annual income less than $15,000 a year? 

() Yes - SKIP TO NEXT SCREENER QUESTION 
() No 

Is your current household's annual income between $15,000 and $35,000 a year? 

() Yes - SKIP TO NEXT SCREENER QUESTION 
() No 


C-12 

PRE-DECISIONAL - FOR INTERNAL IRS W&l USE ONLY - NOT FOR PUBLIC RELEASE 



PRE-DECISIONAL - FOR INTERNAL IRS W&l USE ONLY - NOT FOR PUBLIC RELEASE 

Is your current household's annual income between $35,000 and $50,000 a year? 

() Yes - SKIP TO NEXT SCREENER QUESTION 
()N ° 

Is your current household's annual income between $50,000 and $75,000 a year? 

() Yes - SKIP TO NEXT SCREENER QUESTION 
() No 

Is your current household's annual income between $75,000 and $90,000 a year? 

() Yes - SKIP TO NEXT SCREENER QUESTION 
() No 

Is your current household's annual income more than $90,000 a year? 

() Yes 
() No 

11. What is the highest degree or level of school you have completed? 

( ) No schooling completed 

( ) Completed schooling through Grade_. 

( ) Received High School Diploma, GED or alternative credential for high school graduation 
( ) Associate's Degree 
( ) Bachelor's Degree 

( ) Master's Degree (for example: MA, MS, Meng, MEd, MSW, MBA) 

( ) Professional degree beyond a bachelor's degree (for example: MD, DDS, DVM, LLB, JD) 

( ) Doctorate degree (for example: PhD, EdD) 

12. Did you or your spouse receive military pay in 2018? 

( ) Yes 

( ) No 

( ) Don't know 


Thank you for taking time to answer these questions. Please hold for a moment while I verify your 
eligibility. 

Thank you for waiting. You are eligible to take part in our study. Participation requires that you attend 
an in-person interview at a public location where MITRE researchers are conducting the interviews. At 
the in-person interview, you will receive a $100 VISA prepaid gift card as a thank you for participating in 
this important study. Of the following time slots, which works best for you? [Read list of available time 
slots from tracking spreadsheet, record in spreadsheet and confirm follow up contact information]. 
We will contact you again before the interview to confirm your participation. Have a great day! 


C-13 

PRE-DECISIONAL - FOR INTERNAL IRS W&l USE ONLY - NOT FOR PUBLIC RELEASE 



PRE-DECISIONAL - FOR INTERNAL IRS W&l USE ONLY - NOT FOR PUBLIC RELEASE 


Below are the embedded document that contains the usability testing interview protocol 
described in Section 2.5. 



Usability Testing 
Protocol.docx 
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Appendix E Acronyms 


AGI 

Adjusted Gross Income 

CDW 

Compliance Data Warehouse 

EIN 

Employer Identification Number 

EITC 

Earned Income Tax Credit 

FFA 

Free File Alliance, LLC 

IRS 

Internal Revenue Service 

ISO 

International Organization for Standardization 

MOU 

Memorandum of Understanding 

OMB 

Office of Management and Budget’s 

SEQ 

Single Ease Question 

SUS 

Systems Usability Scale 

TIN 

Taxpayer Identification Number 

TY 

Tax Year 

UMUX 

Usability Metric for User Experience 

W&I 

Wage and Investment 
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